2024-05-08T00:59:41.907 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8
2024-05-08T00:59:41.907 DEBUG:teuthology.run:Teuthology command: teuthology --description rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-policy-none} --owner scheduled_dis@teuthology --name dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi --verbose --archive /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696665 -- /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696665/orig.config.yaml
2024-05-08T00:59:41.953 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-05-08T00:59:42.038 INFO:teuthology.run:Config:
  archive_path: /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696665
  branch: wip-dis-testing-squid
  description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node
    openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid
    supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-policy-none}
  email: idryomov@gmail.com
  first_in_suite: false
  job_id: '7696665'
  kernel:
    kdb: true
    sha1: distro
  ktype: distro
  last_in_suite: false
  machine_type: smithi
  meta:
  - desc: run two ceph clusters and install rbd-mirror
  - desc: configure the permissions for client.mirror
  - desc: configure the permissions for client.mirror
  - desc: 2 ceph clusters with 1 mon and 3 osds each
  - desc: run the rbd_mirror.sh workunit to test the rbd-mirror daemon in journal mode
  name: dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi
  no_nested_subset: false
  nuke-on-error: true
  openstack:
  - volumes:
      count: 3
      size: 30
  os_type: ubuntu
  os_version: '22.04'
  overrides:
    admin_socket:
      branch: wip-dis-testing-squid
    ceph:
      conf:
        client:
          debug rbd: 20
          debug rbd_mirror: 15
          log to stderr: false
          rbd default features: 125
          rbd mirror image policy type: none
        client.mirror.0:
          admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok
          pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid
        client.mirror.1:
          admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok
          pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid
        client.mirror.2:
          admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok
          pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid
        client.mirror.3:
          admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok
          pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid
        client.mirror.4:
          admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok
          pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid
        client.mirror.5:
          admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok
          pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid
        client.mirror.6:
          admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok
          pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid
        global:
          mon client directed command retry: 5
          mon warn on pool no app: false
          ms inject socket failures: 5000
        mgr:
          debug mgr: 20
          debug ms: 1
        mon:
          debug mon: 20
          debug ms: 1
          debug paxos: 20
        osd:
          bluefs allocator: hybrid
          bluestore allocator: hybrid
          bluestore block size: 96636764160
          bluestore fsck on mount: true
          debug bluefs: 1/20
          debug bluestore: 1/20
          debug ms: 1
          debug osd: 20
          debug rocksdb: 4/10
          mon osd backfillfull_ratio: 0.85
          mon osd full ratio: 0.9
          mon osd nearfull ratio: 0.8
          osd failsafe full ratio: 0.95
          osd objectstore: bluestore
      flavor: default
      fs: xfs
      log-ignorelist:
      - \(MDS_ALL_DOWN\)
      - \(MDS_UP_LESS_THAN_MAX\)
      - \(OSD_SLOW_PING_TIME
      sha1: e0c1691aade07c0b78ccff9531f65d3c80e3e220
    ceph-deploy:
      bluestore: true
      conf:
        client:
          log file: /var/log/ceph/ceph-$name.$pid.log
        mon:
          osd default pool size: 2
        osd:
          bluestore block size: 96636764160
          bluestore fsck on mount: true
          debug bluefs: 1/20
          debug bluestore: 1/20
          debug rocksdb: 4/10
          mon osd backfillfull_ratio: 0.85
          mon osd full ratio: 0.9
          mon osd nearfull ratio: 0.8
          osd failsafe full ratio: 0.95
          osd objectstore: bluestore
      fs: xfs
    install:
      ceph:
        flavor: default
        sha1: e0c1691aade07c0b78ccff9531f65d3c80e3e220
    thrashosds:
      bdev_inject_crash: 2
      bdev_inject_crash_probability: 0.5
    workunit:
      branch: wip-dis-testing-squid
      sha1: 9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2
  owner: scheduled_dis@teuthology
  priority: 70
  repo: https://github.com/ceph/ceph-ci
  roles:
  - - cluster1.mon.a
    - cluster1.mgr.x
    - cluster2.mgr.x
    - cluster1.osd.0
    - cluster1.osd.1
    - cluster1.osd.2
    - cluster1.client.0
    - cluster2.client.0
  - - cluster2.mon.a
    - cluster2.osd.0
    - cluster2.osd.1
    - cluster2.osd.2
    - cluster1.client.mirror
    - cluster1.client.mirror.0
    - cluster1.client.mirror.1
    - cluster1.client.mirror.2
    - cluster1.client.mirror.3
    - cluster1.client.mirror.4
    - cluster1.client.mirror.5
    - cluster1.client.mirror.6
    - cluster2.client.mirror
    - cluster2.client.mirror.0
    - cluster2.client.mirror.1
    - cluster2.client.mirror.2
    - cluster2.client.mirror.3
    - cluster2.client.mirror.4
    - cluster2.client.mirror.5
    - cluster2.client.mirror.6
  seed: 5306
  sha1: e0c1691aade07c0b78ccff9531f65d3c80e3e220
  sleep_before_teardown: 0
  subset: 21/99
  suite: rbd
  suite_branch: wip-dis-testing-squid
  suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2/qa
  suite_relpath: qa
  suite_repo: https://github.com/ceph/ceph-ci
  suite_sha1: 9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2
  targets:
    smithi106.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEhs3TLZ8P9r3DOr+8IGGUHcnfAVbx7TqpskSVSFgCjPGwIhD09SIBtp+Z7rjLdnhz1N5Qf4WDMEJUB4B73G1wU=
    smithi142.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAA1toN6R9IHXeOAxGgCT6AY7lk0P4OC4lEcl3mqalcJ3g8GXSNmbOrbouPGsGsi9R87RjctY7iw/lylaW12BKM=
  tasks:
  - install:
      extra_packages:
      - rbd-mirror
  - ceph:
      cluster: cluster1
  - ceph:
      cluster: cluster2
  - exec:
      cluster1.client.mirror.0:
      - sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror'
        osd 'profile rbd'
      - sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror'
        osd 'profile rbd'
      - sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror'
        osd 'profile rbd'
      - sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror'
        osd 'profile rbd'
      - sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror'
        osd 'profile rbd'
      - sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror'
        osd 'profile rbd'
  - exec:
      cluster1.client.mirror.0:
      - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer'
        osd 'profile rbd'
      - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror'
        osd 'profile rbd'
      - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror'
        osd 'profile rbd'
      - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror'
        osd 'profile rbd'
      - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror'
        osd 'profile rbd'
      - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer'
        osd 'profile rbd'
      - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror'
        osd 'profile rbd'
      - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror'
        osd 'profile rbd'
      - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror'
        osd 'profile rbd'
      - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror'
        osd 'profile rbd'
  - workunit:
      clients:
        cluster1.client.mirror:
        - rbd/rbd_mirror.sh
      env:
        CEPH_ARGS: ''
        RBD_MIRROR_INSTANCES: '4'
        RBD_MIRROR_MODE: journal
        RBD_MIRROR_USE_EXISTING_CLUSTER: '1'
  teuthology:
    fragments_dropped: []
    meta: {}
    postmerge: []
  teuthology_branch: main
  teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1
  timestamp: 2024-05-07_23:02:08
  tube: smithi
  user: dis
  verbose: true
  worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885
2024-05-08T00:59:42.038 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2/qa; will attempt to use it
2024-05-08T00:59:42.039 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2/qa/tasks
2024-05-08T00:59:42.039 INFO:teuthology.run_tasks:Running task internal.check_packages...
2024-05-08T00:59:42.040 INFO:teuthology.task.internal:Checking packages...
2024-05-08T00:59:42.065 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'
2024-05-08T00:59:42.065 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch
2024-05-08T00:59:42.065 INFO:teuthology.packaging:ref: None
2024-05-08T00:59:42.065 INFO:teuthology.packaging:tag: None
2024-05-08T00:59:42.065 INFO:teuthology.packaging:branch: wip-dis-testing-squid
2024-05-08T00:59:42.065 INFO:teuthology.packaging:sha1: e0c1691aade07c0b78ccff9531f65d3c80e3e220
2024-05-08T00:59:42.065 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-dis-testing-squid
2024-05-08T00:59:42.216 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2529-ge0c1691a-1jammy
2024-05-08T00:59:42.217 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep...
2024-05-08T00:59:42.227 INFO:teuthology.task.internal:no buildpackages task found
2024-05-08T00:59:42.227 INFO:teuthology.run_tasks:Running task internal.save_config...
2024-05-08T00:59:42.240 INFO:teuthology.task.internal:Saving configuration
2024-05-08T00:59:42.258 INFO:teuthology.run_tasks:Running task internal.check_lock...
2024-05-08T00:59:42.269 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-05-08T00:59:42.293 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696665', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-08 00:54:07.591181', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPjtZG13ONBZ1o+DQUoajBBulZRPeyIiP8FzsJoSThoL'}
2024-05-08T00:59:42.316 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi142.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696665', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-08 00:54:07.589674', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBnxrUV5RD0g9jTPXUyXhuduS4B5+oPXjq6un3skcYNc'}
2024-05-08T00:59:42.316 INFO:teuthology.run_tasks:Running task internal.add_remotes...
2024-05-08T00:59:42.327 INFO:teuthology.task.internal:roles: ubuntu@smithi106.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0']
2024-05-08T00:59:42.327 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']
2024-05-08T00:59:42.328 INFO:teuthology.run_tasks:Running task console_log...
2024-05-08T00:59:42.402 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fb2ffbef790>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fb2ff727310>, signals=[15])
2024-05-08T00:59:42.403 INFO:teuthology.run_tasks:Running task internal.connect...
2024-05-08T00:59:42.412 INFO:teuthology.task.internal:Opening connections...
2024-05-08T00:59:42.412 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi106.front.sepia.ceph.com
2024-05-08T00:59:42.414 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T00:59:42.494 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi142.front.sepia.ceph.com
2024-05-08T00:59:42.495 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T00:59:42.571 INFO:teuthology.run_tasks:Running task internal.push_inventory...
2024-05-08T00:59:42.577 DEBUG:teuthology.orchestra.run.smithi106:> uname -m
2024-05-08T00:59:42.583 INFO:teuthology.orchestra.run.smithi106.stdout:x86_64
2024-05-08T00:59:42.583 DEBUG:teuthology.orchestra.run.smithi106:> cat /etc/os-release
2024-05-08T00:59:42.629 INFO:teuthology.orchestra.run.smithi106.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2024-05-08T00:59:42.629 INFO:teuthology.orchestra.run.smithi106.stdout:NAME="Ubuntu"
2024-05-08T00:59:42.629 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_ID="22.04"
2024-05-08T00:59:42.630 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2024-05-08T00:59:42.630 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_CODENAME=jammy
2024-05-08T00:59:42.630 INFO:teuthology.orchestra.run.smithi106.stdout:ID=ubuntu
2024-05-08T00:59:42.630 INFO:teuthology.orchestra.run.smithi106.stdout:ID_LIKE=debian
2024-05-08T00:59:42.630 INFO:teuthology.orchestra.run.smithi106.stdout:HOME_URL="https://www.ubuntu.com/"
2024-05-08T00:59:42.630 INFO:teuthology.orchestra.run.smithi106.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2024-05-08T00:59:42.630 INFO:teuthology.orchestra.run.smithi106.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2024-05-08T00:59:42.630 INFO:teuthology.orchestra.run.smithi106.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2024-05-08T00:59:42.630 INFO:teuthology.orchestra.run.smithi106.stdout:UBUNTU_CODENAME=jammy
2024-05-08T00:59:42.631 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server
2024-05-08T00:59:42.652 DEBUG:teuthology.orchestra.run.smithi142:> uname -m
2024-05-08T00:59:42.657 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64
2024-05-08T00:59:42.657 DEBUG:teuthology.orchestra.run.smithi142:> cat /etc/os-release
2024-05-08T00:59:42.707 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2024-05-08T00:59:42.708 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="Ubuntu"
2024-05-08T00:59:42.708 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="22.04"
2024-05-08T00:59:42.708 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2024-05-08T00:59:42.708 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_CODENAME=jammy
2024-05-08T00:59:42.708 INFO:teuthology.orchestra.run.smithi142.stdout:ID=ubuntu
2024-05-08T00:59:42.708 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE=debian
2024-05-08T00:59:42.708 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://www.ubuntu.com/"
2024-05-08T00:59:42.708 INFO:teuthology.orchestra.run.smithi142.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2024-05-08T00:59:42.708 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2024-05-08T00:59:42.708 INFO:teuthology.orchestra.run.smithi142.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2024-05-08T00:59:42.708 INFO:teuthology.orchestra.run.smithi142.stdout:UBUNTU_CODENAME=jammy
2024-05-08T00:59:42.709 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server
2024-05-08T00:59:42.727 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles...
2024-05-08T00:59:42.742 INFO:teuthology.run_tasks:Running task internal.check_conflict...
2024-05-08T00:59:42.752 INFO:teuthology.task.internal:Checking for old test directory...
2024-05-08T00:59:42.753 DEBUG:teuthology.orchestra.run.smithi106:> test '!' -e /home/ubuntu/cephtest
2024-05-08T00:59:42.755 DEBUG:teuthology.orchestra.run.smithi142:> test '!' -e /home/ubuntu/cephtest
2024-05-08T00:59:42.760 INFO:teuthology.run_tasks:Running task internal.check_ceph_data...
2024-05-08T00:59:42.771 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph...
2024-05-08T00:59:42.771 DEBUG:teuthology.orchestra.run.smithi106:> test -z $(ls -A /var/lib/ceph)
2024-05-08T00:59:42.803 DEBUG:teuthology.orchestra.run.smithi142:> test -z $(ls -A /var/lib/ceph)
2024-05-08T00:59:42.811 INFO:teuthology.run_tasks:Running task internal.vm_setup...
2024-05-08T00:59:42.868 INFO:teuthology.run_tasks:Running task kernel...
2024-05-08T00:59:42.885 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'}
2024-05-08T00:59:42.886 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300
2024-05-08T00:59:42.886 DEBUG:teuthology.orchestra.run.smithi106:> test -f /run/.containerenv -o -f /.dockerenv
2024-05-08T00:59:42.886 DEBUG:teuthology.orchestra.run.smithi142:> test -f /run/.containerenv -o -f /.dockerenv
2024-05-08T00:59:42.890 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T00:59:42.891 DEBUG:teuthology.orchestra.run.smithi106:> uname -r
2024-05-08T00:59:42.891 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T00:59:42.892 DEBUG:teuthology.orchestra.run.smithi142:> uname -r
2024-05-08T00:59:42.938 INFO:teuthology.orchestra.run.smithi106.stdout:5.15.0-56-generic
2024-05-08T00:59:42.938 INFO:teuthology.task.kernel:Running kernel on smithi106: 5.15.0-56-generic
2024-05-08T00:59:42.938 DEBUG:teuthology.orchestra.run.smithi106:> sudo apt-get clean
2024-05-08T00:59:42.941 INFO:teuthology.orchestra.run.smithi142.stdout:5.15.0-56-generic
2024-05-08T00:59:42.942 INFO:teuthology.task.kernel:Running kernel on smithi142: 5.15.0-56-generic
2024-05-08T00:59:42.942 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get clean
2024-05-08T00:59:43.062 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get update
2024-05-08T00:59:43.072 DEBUG:teuthology.orchestra.run.smithi106:> sudo apt-get update
2024-05-08T00:59:43.238 INFO:teuthology.orchestra.run.smithi142.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
2024-05-08T00:59:43.304 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease
2024-05-08T00:59:43.309 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease
2024-05-08T00:59:43.314 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
2024-05-08T00:59:43.330 INFO:teuthology.orchestra.run.smithi106.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
2024-05-08T00:59:43.330 INFO:teuthology.orchestra.run.smithi106.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
2024-05-08T00:59:43.409 INFO:teuthology.orchestra.run.smithi106.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB]
2024-05-08T00:59:43.416 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB]
2024-05-08T00:59:43.540 INFO:teuthology.orchestra.run.smithi106.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,638 kB]
2024-05-08T00:59:43.564 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [458 kB]
2024-05-08T00:59:43.639 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,638 kB]
2024-05-08T00:59:43.653 INFO:teuthology.orchestra.run.smithi106.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [624 kB]
2024-05-08T00:59:43.656 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,421 kB]
2024-05-08T00:59:43.665 INFO:teuthology.orchestra.run.smithi106.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [307 kB]
2024-05-08T00:59:43.672 INFO:teuthology.orchestra.run.smithi106.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB]
2024-05-08T00:59:43.672 INFO:teuthology.orchestra.run.smithi106.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB]
2024-05-08T00:59:43.673 INFO:teuthology.orchestra.run.smithi106.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,864 kB]
2024-05-08T00:59:43.709 INFO:teuthology.orchestra.run.smithi106.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [316 kB]
2024-05-08T00:59:43.715 INFO:teuthology.orchestra.run.smithi106.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B]
2024-05-08T00:59:43.715 INFO:teuthology.orchestra.run.smithi106.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [701 kB]
2024-05-08T00:59:43.729 INFO:teuthology.orchestra.run.smithi106.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,072 kB]
2024-05-08T00:59:43.742 INFO:teuthology.orchestra.run.smithi106.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [458 kB]
2024-05-08T00:59:43.745 INFO:teuthology.orchestra.run.smithi106.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [245 kB]
2024-05-08T00:59:43.749 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [246 kB]
2024-05-08T00:59:43.750 INFO:teuthology.orchestra.run.smithi106.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB]
2024-05-08T00:59:43.754 INFO:teuthology.orchestra.run.smithi106.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB]
2024-05-08T00:59:43.755 INFO:teuthology.orchestra.run.smithi142.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB]
2024-05-08T00:59:43.757 INFO:teuthology.orchestra.run.smithi142.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,807 kB]
2024-05-08T00:59:43.776 INFO:teuthology.orchestra.run.smithi106.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B]
2024-05-08T00:59:43.776 INFO:teuthology.orchestra.run.smithi106.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB]
2024-05-08T00:59:43.780 INFO:teuthology.orchestra.run.smithi106.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B]
2024-05-08T00:59:43.780 INFO:teuthology.orchestra.run.smithi106.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB]
2024-05-08T00:59:43.780 INFO:teuthology.orchestra.run.smithi106.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB]
2024-05-08T00:59:43.782 INFO:teuthology.orchestra.run.smithi106.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB]
2024-05-08T00:59:43.782 INFO:teuthology.orchestra.run.smithi106.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B]
2024-05-08T00:59:43.784 INFO:teuthology.orchestra.run.smithi106.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB]
2024-05-08T00:59:43.785 INFO:teuthology.orchestra.run.smithi106.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB]
2024-05-08T00:59:43.800 INFO:teuthology.orchestra.run.smithi142.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB]
2024-05-08T00:59:43.801 INFO:teuthology.orchestra.run.smithi142.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [307 kB]
2024-05-08T00:59:43.806 INFO:teuthology.orchestra.run.smithi142.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B]
2024-05-08T00:59:43.806 INFO:teuthology.orchestra.run.smithi142.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [602 kB]
2024-05-08T00:59:43.815 INFO:teuthology.orchestra.run.smithi106.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB]
2024-05-08T00:59:43.815 INFO:teuthology.orchestra.run.smithi142.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [849 kB]
2024-05-08T00:59:43.816 INFO:teuthology.orchestra.run.smithi106.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B]
2024-05-08T00:59:43.826 INFO:teuthology.orchestra.run.smithi142.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [624 kB]
2024-05-08T00:59:43.831 INFO:teuthology.orchestra.run.smithi142.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB]
2024-05-08T00:59:43.859 INFO:teuthology.orchestra.run.smithi142.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [307 kB]
2024-05-08T00:59:43.859 INFO:teuthology.orchestra.run.smithi142.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB]
2024-05-08T00:59:43.860 INFO:teuthology.orchestra.run.smithi142.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B]
2024-05-08T00:59:43.860 INFO:teuthology.orchestra.run.smithi142.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB]
2024-05-08T00:59:43.861 INFO:teuthology.orchestra.run.smithi142.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B]
2024-05-08T00:59:43.861 INFO:teuthology.orchestra.run.smithi142.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B]
2024-05-08T00:59:43.865 INFO:teuthology.orchestra.run.smithi142.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB]
2024-05-08T00:59:43.865 INFO:teuthology.orchestra.run.smithi142.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,864 kB]
2024-05-08T00:59:43.904 INFO:teuthology.orchestra.run.smithi142.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB]
2024-05-08T00:59:43.905 INFO:teuthology.orchestra.run.smithi142.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [316 kB]
2024-05-08T00:59:43.911 INFO:teuthology.orchestra.run.smithi142.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B]
2024-05-08T00:59:43.912 INFO:teuthology.orchestra.run.smithi142.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [701 kB]
2024-05-08T00:59:43.922 INFO:teuthology.orchestra.run.smithi142.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,072 kB]
2024-05-08T00:59:43.939 INFO:teuthology.orchestra.run.smithi142.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [245 kB]
2024-05-08T00:59:43.942 INFO:teuthology.orchestra.run.smithi142.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB]
2024-05-08T00:59:43.942 INFO:teuthology.orchestra.run.smithi142.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B]
2024-05-08T00:59:43.943 INFO:teuthology.orchestra.run.smithi142.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB]
2024-05-08T00:59:43.951 INFO:teuthology.orchestra.run.smithi142.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB]
2024-05-08T00:59:43.951 INFO:teuthology.orchestra.run.smithi142.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B]
2024-05-08T00:59:43.951 INFO:teuthology.orchestra.run.smithi142.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB]
2024-05-08T00:59:43.952 INFO:teuthology.orchestra.run.smithi142.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB]
2024-05-08T00:59:43.953 INFO:teuthology.orchestra.run.smithi142.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB]
2024-05-08T00:59:43.973 INFO:teuthology.orchestra.run.smithi142.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B]
2024-05-08T00:59:43.974 INFO:teuthology.orchestra.run.smithi142.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB]
2024-05-08T00:59:43.974 INFO:teuthology.orchestra.run.smithi142.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB]
2024-05-08T00:59:43.975 INFO:teuthology.orchestra.run.smithi142.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB]
2024-05-08T00:59:43.975 INFO:teuthology.orchestra.run.smithi142.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B]
2024-05-08T00:59:44.026 INFO:teuthology.orchestra.run.smithi106.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,421 kB]
2024-05-08T00:59:44.195 INFO:teuthology.orchestra.run.smithi106.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [246 kB]
2024-05-08T00:59:44.207 INFO:teuthology.orchestra.run.smithi106.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB]
2024-05-08T00:59:44.207 INFO:teuthology.orchestra.run.smithi106.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB]
2024-05-08T00:59:44.209 INFO:teuthology.orchestra.run.smithi106.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,807 kB]
2024-05-08T00:59:44.290 INFO:teuthology.orchestra.run.smithi106.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [307 kB]
2024-05-08T00:59:44.299 INFO:teuthology.orchestra.run.smithi106.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B]
2024-05-08T00:59:44.299 INFO:teuthology.orchestra.run.smithi106.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [849 kB]
2024-05-08T00:59:44.320 INFO:teuthology.orchestra.run.smithi106.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [602 kB]
2024-05-08T00:59:44.496 INFO:teuthology.orchestra.run.smithi106.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB]
2024-05-08T00:59:44.903 INFO:teuthology.orchestra.run.smithi106.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB]
2024-05-08T00:59:44.908 INFO:teuthology.orchestra.run.smithi106.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB]
2024-05-08T00:59:44.922 INFO:teuthology.orchestra.run.smithi106.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B]
2024-05-08T00:59:44.922 INFO:teuthology.orchestra.run.smithi106.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B]
2024-05-08T00:59:44.925 INFO:teuthology.orchestra.run.smithi106.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B]
2024-05-08T00:59:49.414 INFO:teuthology.orchestra.run.smithi106.stdout:Fetched 13.4 MB in 2s (6,803 kB/s)
2024-05-08T00:59:49.520 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 13.4 MB in 2s (6,586 kB/s)
2024-05-08T00:59:50.970 INFO:teuthology.orchestra.run.smithi106.stdout:Reading package lists...
2024-05-08T00:59:50.988 DEBUG:teuthology.orchestra.run.smithi106:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic
2024-05-08T00:59:51.019 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists...
2024-05-08T00:59:51.035 DEBUG:teuthology.orchestra.run.smithi142:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic
2024-05-08T00:59:51.043 INFO:teuthology.orchestra.run.smithi106.stdout:Reading package lists...
2024-05-08T00:59:51.088 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists...
2024-05-08T00:59:51.242 INFO:teuthology.orchestra.run.smithi106.stdout:Building dependency tree...
2024-05-08T00:59:51.243 INFO:teuthology.orchestra.run.smithi106.stdout:Reading state information...
2024-05-08T00:59:51.266 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree...
2024-05-08T00:59:51.267 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information...
2024-05-08T00:59:51.405 INFO:teuthology.orchestra.run.smithi106.stdout:The following packages were automatically installed and are no longer required:
2024-05-08T00:59:51.405 INFO:teuthology.orchestra.run.smithi106.stdout:  libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0
2024-05-08T00:59:51.405 INFO:teuthology.orchestra.run.smithi106.stdout:  libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib
2024-05-08T00:59:51.405 INFO:teuthology.orchestra.run.smithi106.stdout:  libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib
2024-05-08T00:59:51.406 INFO:teuthology.orchestra.run.smithi106.stdout:  python2 python2-dev python2-minimal python2.7 python2.7-dev
2024-05-08T00:59:51.406 INFO:teuthology.orchestra.run.smithi106.stdout:  python2.7-minimal
2024-05-08T00:59:51.406 INFO:teuthology.orchestra.run.smithi106.stdout:Use 'sudo apt autoremove' to remove them.
2024-05-08T00:59:51.407 INFO:teuthology.orchestra.run.smithi106.stdout:The following additional packages will be installed:
2024-05-08T00:59:51.407 INFO:teuthology.orchestra.run.smithi106.stdout:  linux-generic linux-headers-5.15.0-106 linux-headers-5.15.0-106-generic
2024-05-08T00:59:51.407 INFO:teuthology.orchestra.run.smithi106.stdout:  linux-headers-generic linux-image-5.15.0-106-generic
2024-05-08T00:59:51.407 INFO:teuthology.orchestra.run.smithi106.stdout:  linux-modules-5.15.0-106-generic linux-modules-extra-5.15.0-106-generic
2024-05-08T00:59:51.408 INFO:teuthology.orchestra.run.smithi106.stdout:Suggested packages:
2024-05-08T00:59:51.408 INFO:teuthology.orchestra.run.smithi106.stdout:  fdutils linux-doc | linux-source-5.15.0 linux-tools
2024-05-08T00:59:51.408 INFO:teuthology.orchestra.run.smithi106.stdout:Recommended packages:
2024-05-08T00:59:51.409 INFO:teuthology.orchestra.run.smithi106.stdout:  thermald
2024-05-08T00:59:51.424 INFO:teuthology.orchestra.run.smithi142.stdout:The following packages were automatically installed and are no longer required:
2024-05-08T00:59:51.424 INFO:teuthology.orchestra.run.smithi142.stdout:  libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0
2024-05-08T00:59:51.424 INFO:teuthology.orchestra.run.smithi142.stdout:  libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib
2024-05-08T00:59:51.425 INFO:teuthology.orchestra.run.smithi142.stdout:  libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib
2024-05-08T00:59:51.425 INFO:teuthology.orchestra.run.smithi142.stdout:  python2 python2-dev python2-minimal python2.7 python2.7-dev
2024-05-08T00:59:51.425 INFO:teuthology.orchestra.run.smithi142.stdout:  python2.7-minimal
2024-05-08T00:59:51.425 INFO:teuthology.orchestra.run.smithi142.stdout:Use 'sudo apt autoremove' to remove them.
2024-05-08T00:59:51.426 INFO:teuthology.orchestra.run.smithi142.stdout:The following additional packages will be installed:
2024-05-08T00:59:51.426 INFO:teuthology.orchestra.run.smithi142.stdout:  linux-generic linux-headers-5.15.0-106 linux-headers-5.15.0-106-generic
2024-05-08T00:59:51.426 INFO:teuthology.orchestra.run.smithi142.stdout:  linux-headers-generic linux-image-5.15.0-106-generic
2024-05-08T00:59:51.427 INFO:teuthology.orchestra.run.smithi142.stdout:  linux-modules-5.15.0-106-generic linux-modules-extra-5.15.0-106-generic
2024-05-08T00:59:51.428 INFO:teuthology.orchestra.run.smithi142.stdout:Suggested packages:
2024-05-08T00:59:51.428 INFO:teuthology.orchestra.run.smithi142.stdout:  fdutils linux-doc | linux-source-5.15.0 linux-tools
2024-05-08T00:59:51.428 INFO:teuthology.orchestra.run.smithi142.stdout:Recommended packages:
2024-05-08T00:59:51.428 INFO:teuthology.orchestra.run.smithi142.stdout:  thermald
2024-05-08T00:59:51.443 INFO:teuthology.orchestra.run.smithi106.stdout:The following NEW packages will be installed:
2024-05-08T00:59:51.444 INFO:teuthology.orchestra.run.smithi106.stdout:  linux-headers-5.15.0-106 linux-headers-5.15.0-106-generic
2024-05-08T00:59:51.444 INFO:teuthology.orchestra.run.smithi106.stdout:  linux-image-5.15.0-106-generic linux-modules-5.15.0-106-generic
2024-05-08T00:59:51.444 INFO:teuthology.orchestra.run.smithi106.stdout:  linux-modules-extra-5.15.0-106-generic
2024-05-08T00:59:51.445 INFO:teuthology.orchestra.run.smithi106.stdout:The following packages will be upgraded:
2024-05-08T00:59:51.446 INFO:teuthology.orchestra.run.smithi106.stdout:  linux-generic linux-headers-generic linux-image-generic
2024-05-08T00:59:51.482 INFO:teuthology.orchestra.run.smithi142.stdout:The following NEW packages will be installed:
2024-05-08T00:59:51.482 INFO:teuthology.orchestra.run.smithi142.stdout:  linux-headers-5.15.0-106 linux-headers-5.15.0-106-generic
2024-05-08T00:59:51.482 INFO:teuthology.orchestra.run.smithi142.stdout:  linux-image-5.15.0-106-generic linux-modules-5.15.0-106-generic
2024-05-08T00:59:51.483 INFO:teuthology.orchestra.run.smithi142.stdout:  linux-modules-extra-5.15.0-106-generic
2024-05-08T00:59:51.484 INFO:teuthology.orchestra.run.smithi142.stdout:The following packages will be upgraded:
2024-05-08T00:59:51.484 INFO:teuthology.orchestra.run.smithi142.stdout:  linux-generic linux-headers-generic linux-image-generic
2024-05-08T00:59:51.625 INFO:teuthology.orchestra.run.smithi106.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded.
2024-05-08T00:59:51.625 INFO:teuthology.orchestra.run.smithi106.stdout:Need to get 113 MB of archives.
2024-05-08T00:59:51.625 INFO:teuthology.orchestra.run.smithi106.stdout:After this operation, 583 MB of additional disk space will be used.
2024-05-08T00:59:51.626 INFO:teuthology.orchestra.run.smithi106.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-106-generic amd64 5.15.0-106.116 [22.7 MB]
2024-05-08T00:59:51.669 INFO:teuthology.orchestra.run.smithi142.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded.
2024-05-08T00:59:51.669 INFO:teuthology.orchestra.run.smithi142.stdout:Need to get 113 MB of archives.
2024-05-08T00:59:51.669 INFO:teuthology.orchestra.run.smithi142.stdout:After this operation, 583 MB of additional disk space will be used.
2024-05-08T00:59:51.669 INFO:teuthology.orchestra.run.smithi142.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-106-generic amd64 5.15.0-106.116 [22.7 MB]
2024-05-08T00:59:53.119 INFO:teuthology.orchestra.run.smithi106.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-106-generic amd64 5.15.0-106.116 [11.5 MB]
2024-05-08T00:59:53.470 INFO:teuthology.orchestra.run.smithi106.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-106-generic amd64 5.15.0-106.116 [63.9 MB]
2024-05-08T00:59:55.761 INFO:teuthology.orchestra.run.smithi106.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.106.106 [1,692 B]
2024-05-08T00:59:55.761 INFO:teuthology.orchestra.run.smithi106.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.106.106 [2,502 B]
2024-05-08T00:59:55.761 INFO:teuthology.orchestra.run.smithi106.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-106 all 5.15.0-106.116 [12.3 MB]
2024-05-08T00:59:56.114 INFO:teuthology.orchestra.run.smithi106.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-106-generic amd64 5.15.0-106.116 [2,876 kB]
2024-05-08T00:59:56.201 INFO:teuthology.orchestra.run.smithi106.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.106.106 [2,366 B]
2024-05-08T00:59:56.593 INFO:teuthology.orchestra.run.smithi106.stdout:Fetched 113 MB in 5s (23.9 MB/s)
2024-05-08T00:59:56.763 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package linux-modules-5.15.0-106-generic.
2024-05-08T00:59:58.797 INFO:teuthology.orchestra.run.smithi106.stdout:(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 129542 files and directories currently installed.)
2024-05-08T00:59:58.801 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../0-linux-modules-5.15.0-106-generic_5.15.0-106.116_amd64.deb ...
2024-05-08T00:59:58.877 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking linux-modules-5.15.0-106-generic (5.15.0-106.116) ...
2024-05-08T01:00:01.252 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package linux-image-5.15.0-106-generic.
2024-05-08T01:00:01.273 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../1-linux-image-5.15.0-106-generic_5.15.0-106.116_amd64.deb ...
2024-05-08T01:00:01.352 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking linux-image-5.15.0-106-generic (5.15.0-106.116) ...
2024-05-08T01:00:01.847 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-106-generic.
2024-05-08T01:00:01.867 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-106-generic_5.15.0-106.116_amd64.deb ...
2024-05-08T01:00:01.905 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking linux-modules-extra-5.15.0-106-generic (5.15.0-106.116) ...
2024-05-08T01:00:08.020 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../3-linux-generic_5.15.0.106.106_amd64.deb ...
2024-05-08T01:00:08.146 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking linux-generic (5.15.0.106.106) over (5.15.0.56.54) ...
2024-05-08T01:00:08.487 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.106.106_amd64.deb ...
2024-05-08T01:00:08.591 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking linux-image-generic (5.15.0.106.106) over (5.15.0.56.54) ...
2024-05-08T01:00:08.869 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package linux-headers-5.15.0-106.
2024-05-08T01:00:08.891 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../5-linux-headers-5.15.0-106_5.15.0-106.116_all.deb ...
2024-05-08T01:00:08.935 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking linux-headers-5.15.0-106 (5.15.0-106.116) ...
2024-05-08T01:00:14.564 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package linux-headers-5.15.0-106-generic.
2024-05-08T01:00:14.600 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../6-linux-headers-5.15.0-106-generic_5.15.0-106.116_amd64.deb ...
2024-05-08T01:00:14.638 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking linux-headers-5.15.0-106-generic (5.15.0-106.116) ...
2024-05-08T01:00:17.184 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.106.106_amd64.deb ...
2024-05-08T01:00:17.307 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking linux-headers-generic (5.15.0.106.106) over (5.15.0.56.54) ...
2024-05-08T01:00:17.622 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up linux-headers-5.15.0-106 (5.15.0-106.116) ...
2024-05-08T01:00:17.745 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up linux-headers-5.15.0-106-generic (5.15.0-106.116) ...
2024-05-08T01:00:17.871 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up linux-headers-generic (5.15.0.106.106) ...
2024-05-08T01:00:17.999 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up linux-image-5.15.0-106-generic (5.15.0-106.116) ...
2024-05-08T01:00:19.371 INFO:teuthology.orchestra.run.smithi106.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic
2024-05-08T01:00:19.371 INFO:teuthology.orchestra.run.smithi106.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic
2024-05-08T01:00:19.371 INFO:teuthology.orchestra.run.smithi106.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-106-generic
2024-05-08T01:00:19.371 INFO:teuthology.orchestra.run.smithi106.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-106-generic
2024-05-08T01:00:19.609 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up linux-modules-5.15.0-106-generic (5.15.0-106.116) ...
2024-05-08T01:00:20.994 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up linux-modules-extra-5.15.0-106-generic (5.15.0-106.116) ...
2024-05-08T01:00:22.395 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up linux-image-generic (5.15.0.106.106) ...
2024-05-08T01:00:22.504 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up linux-generic (5.15.0.106.106) ...
2024-05-08T01:00:22.614 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for linux-image-5.15.0-106-generic (5.15.0-106.116) ...
2024-05-08T01:00:22.674 INFO:teuthology.orchestra.run.smithi106.stdout:/etc/kernel/postinst.d/initramfs-tools:
2024-05-08T01:00:22.674 INFO:teuthology.orchestra.run.smithi106.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-106-generic
2024-05-08T01:00:36.650 INFO:teuthology.orchestra.run.smithi106.stdout:/etc/kernel/postinst.d/zz-update-grub:
2024-05-08T01:00:36.650 INFO:teuthology.orchestra.run.smithi106.stdout:Sourcing file `/etc/default/grub'
2024-05-08T01:00:36.668 INFO:teuthology.orchestra.run.smithi106.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg'
2024-05-08T01:00:36.692 INFO:teuthology.orchestra.run.smithi106.stdout:Generating grub configuration file ...
2024-05-08T01:00:36.960 INFO:teuthology.orchestra.run.smithi106.stdout:Found linux image: /boot/vmlinuz-5.15.0-106-generic
2024-05-08T01:00:36.980 INFO:teuthology.orchestra.run.smithi106.stdout:Found initrd image: /boot/initrd.img-5.15.0-106-generic
2024-05-08T01:00:37.288 INFO:teuthology.orchestra.run.smithi106.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic
2024-05-08T01:00:37.293 INFO:teuthology.orchestra.run.smithi106.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic
2024-05-08T01:00:37.368 INFO:teuthology.orchestra.run.smithi106.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic
2024-05-08T01:00:37.374 INFO:teuthology.orchestra.run.smithi106.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic
2024-05-08T01:00:37.530 INFO:teuthology.orchestra.run.smithi106.stdout:Warning: os-prober will not be executed to detect other bootable partitions.
2024-05-08T01:00:37.531 INFO:teuthology.orchestra.run.smithi106.stdout:Systems on them will not be added to the GRUB boot configuration.
2024-05-08T01:00:37.531 INFO:teuthology.orchestra.run.smithi106.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry.
2024-05-08T01:00:37.569 INFO:teuthology.orchestra.run.smithi106.stdout:done
2024-05-08T01:00:38.101 DEBUG:teuthology.orchestra.run.smithi106:> dpkg -s linux-image-generic
2024-05-08T01:00:38.166 INFO:teuthology.orchestra.run.smithi106.stdout:Package: linux-image-generic
2024-05-08T01:00:38.167 INFO:teuthology.orchestra.run.smithi106.stdout:Status: install ok installed
2024-05-08T01:00:38.167 INFO:teuthology.orchestra.run.smithi106.stdout:Priority: optional
2024-05-08T01:00:38.167 INFO:teuthology.orchestra.run.smithi106.stdout:Section: kernel
2024-05-08T01:00:38.167 INFO:teuthology.orchestra.run.smithi106.stdout:Installed-Size: 21
2024-05-08T01:00:38.167 INFO:teuthology.orchestra.run.smithi106.stdout:Maintainer: Ubuntu Kernel Team <kernel-team@lists.ubuntu.com>
2024-05-08T01:00:38.167 INFO:teuthology.orchestra.run.smithi106.stdout:Architecture: amd64
2024-05-08T01:00:38.167 INFO:teuthology.orchestra.run.smithi106.stdout:Source: linux-meta
2024-05-08T01:00:38.167 INFO:teuthology.orchestra.run.smithi106.stdout:Version: 5.15.0.106.106
2024-05-08T01:00:38.167 INFO:teuthology.orchestra.run.smithi106.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-106), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3)
2024-05-08T01:00:38.167 INFO:teuthology.orchestra.run.smithi106.stdout:Depends: linux-image-5.15.0-106-generic, linux-modules-extra-5.15.0-106-generic, linux-firmware, intel-microcode, amd64-microcode
2024-05-08T01:00:38.167 INFO:teuthology.orchestra.run.smithi106.stdout:Recommends: thermald
2024-05-08T01:00:38.167 INFO:teuthology.orchestra.run.smithi106.stdout:Description: Generic Linux kernel image
2024-05-08T01:00:38.167 INFO:teuthology.orchestra.run.smithi106.stdout: This package will always depend on the latest generic kernel image
2024-05-08T01:00:38.167 INFO:teuthology.orchestra.run.smithi106.stdout: available.
2024-05-08T01:00:38.168 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-106-generic
2024-05-08T01:00:38.168 INFO:teuthology.task.kernel:Skipping firmware on distro kernel
2024-05-08T01:00:38.168 DEBUG:teuthology.task.kernel:src is distro, skipping download
2024-05-08T01:00:38.168 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a...
2024-05-08T01:00:38.168 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi106.front.sepia.ceph.com, path=None, version=distro)
2024-05-08T01:00:38.168 DEBUG:teuthology.orchestra.run.smithi106:> sudo apt-get clean
2024-05-08T01:00:38.349 DEBUG:teuthology.orchestra.run.smithi106:> sudo apt-get update
2024-05-08T01:00:38.473 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease
2024-05-08T01:00:38.474 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease
2024-05-08T01:00:38.486 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
2024-05-08T01:00:38.507 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
2024-05-08T01:00:39.870 INFO:teuthology.orchestra.run.smithi106.stdout:Reading package lists...
2024-05-08T01:00:39.887 DEBUG:teuthology.orchestra.run.smithi106:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic
2024-05-08T01:00:39.951 INFO:teuthology.orchestra.run.smithi106.stdout:Reading package lists...
2024-05-08T01:00:40.154 INFO:teuthology.orchestra.run.smithi106.stdout:Building dependency tree...
2024-05-08T01:00:40.154 INFO:teuthology.orchestra.run.smithi106.stdout:Reading state information...
2024-05-08T01:00:40.333 INFO:teuthology.orchestra.run.smithi106.stdout:linux-image-generic is already the newest version (5.15.0.106.106).
2024-05-08T01:00:40.333 INFO:teuthology.orchestra.run.smithi106.stdout:The following packages were automatically installed and are no longer required:
2024-05-08T01:00:40.333 INFO:teuthology.orchestra.run.smithi106.stdout:  libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0
2024-05-08T01:00:40.333 INFO:teuthology.orchestra.run.smithi106.stdout:  libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib
2024-05-08T01:00:40.333 INFO:teuthology.orchestra.run.smithi106.stdout:  libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib
2024-05-08T01:00:40.333 INFO:teuthology.orchestra.run.smithi106.stdout:  linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2
2024-05-08T01:00:40.334 INFO:teuthology.orchestra.run.smithi106.stdout:  python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal
2024-05-08T01:00:40.334 INFO:teuthology.orchestra.run.smithi106.stdout:Use 'sudo apt autoremove' to remove them.
2024-05-08T01:00:40.375 INFO:teuthology.orchestra.run.smithi106.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded.
2024-05-08T01:00:40.377 DEBUG:teuthology.orchestra.run.smithi106:> dpkg -s linux-image-generic
2024-05-08T01:00:40.393 INFO:teuthology.orchestra.run.smithi106.stdout:Package: linux-image-generic
2024-05-08T01:00:40.393 INFO:teuthology.orchestra.run.smithi106.stdout:Status: install ok installed
2024-05-08T01:00:40.393 INFO:teuthology.orchestra.run.smithi106.stdout:Priority: optional
2024-05-08T01:00:40.393 INFO:teuthology.orchestra.run.smithi106.stdout:Section: kernel
2024-05-08T01:00:40.393 INFO:teuthology.orchestra.run.smithi106.stdout:Installed-Size: 21
2024-05-08T01:00:40.393 INFO:teuthology.orchestra.run.smithi106.stdout:Maintainer: Ubuntu Kernel Team <kernel-team@lists.ubuntu.com>
2024-05-08T01:00:40.393 INFO:teuthology.orchestra.run.smithi106.stdout:Architecture: amd64
2024-05-08T01:00:40.394 INFO:teuthology.orchestra.run.smithi106.stdout:Source: linux-meta
2024-05-08T01:00:40.394 INFO:teuthology.orchestra.run.smithi106.stdout:Version: 5.15.0.106.106
2024-05-08T01:00:40.394 INFO:teuthology.orchestra.run.smithi106.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-106), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3)
2024-05-08T01:00:40.394 INFO:teuthology.orchestra.run.smithi106.stdout:Depends: linux-image-5.15.0-106-generic, linux-modules-extra-5.15.0-106-generic, linux-firmware, intel-microcode, amd64-microcode
2024-05-08T01:00:40.394 INFO:teuthology.orchestra.run.smithi106.stdout:Recommends: thermald
2024-05-08T01:00:40.394 INFO:teuthology.orchestra.run.smithi106.stdout:Description: Generic Linux kernel image
2024-05-08T01:00:40.394 INFO:teuthology.orchestra.run.smithi106.stdout: This package will always depend on the latest generic kernel image
2024-05-08T01:00:40.394 INFO:teuthology.orchestra.run.smithi106.stdout: available.
2024-05-08T01:00:40.394 DEBUG:teuthology.orchestra.run.smithi106:> mktemp
2024-05-08T01:00:40.440 INFO:teuthology.orchestra.run.smithi106.stdout:/tmp/tmp.QlXl7nWH1a
2024-05-08T01:00:40.440 DEBUG:teuthology.orchestra.run.smithi106:> sudo cp /boot/grub/grub.cfg /tmp/tmp.QlXl7nWH1a
2024-05-08T01:00:40.496 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0666 /tmp/tmp.QlXl7nWH1a
2024-05-08T01:00:40.618 DEBUG:teuthology.orchestra.remote:smithi106:/tmp/tmp.QlXl7nWH1a is 10KB
2024-05-08T01:00:40.631 DEBUG:teuthology.orchestra.run.smithi106:> rm -fr /tmp/tmp.QlXl7nWH1a
2024-05-08T01:00:40.637 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /etc/grub.d/01_ceph_kernel
2024-05-08T01:00:40.696 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:00:40.696 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/grub.d/01_ceph_kernel
2024-05-08T01:00:40.696 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 755 /etc/grub.d/01_ceph_kernel
2024-05-08T01:00:40.762 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-106-generic
2024-05-08T01:00:40.762 DEBUG:teuthology.orchestra.run.smithi106:> sudo update-grub
2024-05-08T01:00:41.959 INFO:teuthology.orchestra.run.smithi106.stderr:Sourcing file `/etc/default/grub'
2024-05-08T01:00:41.960 INFO:teuthology.orchestra.run.smithi106.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg'
2024-05-08T01:00:41.962 INFO:teuthology.orchestra.run.smithi106.stderr:Generating grub configuration file ...
2024-05-08T01:00:42.154 INFO:teuthology.orchestra.run.smithi106.stderr:Found linux image: /boot/vmlinuz-5.15.0-106-generic
2024-05-08T01:00:42.161 INFO:teuthology.orchestra.run.smithi106.stderr:Found initrd image: /boot/initrd.img-5.15.0-106-generic
2024-05-08T01:00:42.398 INFO:teuthology.orchestra.run.smithi106.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic
2024-05-08T01:00:42.403 INFO:teuthology.orchestra.run.smithi106.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic
2024-05-08T01:00:42.446 INFO:teuthology.orchestra.run.smithi106.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic
2024-05-08T01:00:42.452 INFO:teuthology.orchestra.run.smithi106.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic
2024-05-08T01:00:42.569 INFO:teuthology.orchestra.run.smithi106.stderr:Warning: os-prober will not be executed to detect other bootable partitions.
2024-05-08T01:00:42.569 INFO:teuthology.orchestra.run.smithi106.stderr:Systems on them will not be added to the GRUB boot configuration.
2024-05-08T01:00:42.569 INFO:teuthology.orchestra.run.smithi106.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry.
2024-05-08T01:00:42.598 INFO:teuthology.orchestra.run.smithi106.stderr:done
2024-05-08T01:00:42.603 DEBUG:teuthology.orchestra.run.smithi106:> sudo shutdown -r now
2024-05-08T01:01:11.507 INFO:teuthology.orchestra.run.smithi142.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-106-generic amd64 5.15.0-106.116 [11.5 MB]
2024-05-08T01:01:11.857 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-106-generic amd64 5.15.0-106.116 [63.9 MB]
2024-05-08T01:01:12.606 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version...
2024-05-08T01:01:12.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com'
2024-05-08T01:01:12.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T01:01:13.152 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.106.106 [1,692 B]
2024-05-08T01:01:13.173 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.106.106 [2,502 B]
2024-05-08T01:01:13.194 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-106 all 5.15.0-106.116 [12.3 MB]
2024-05-08T01:01:13.404 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-106-generic amd64 5.15.0-106.116 [2,876 kB]
2024-05-08T01:01:13.478 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.106.106 [2,366 B]
2024-05-08T01:01:13.869 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 113 MB in 1min 22s (1,383 kB/s)
2024-05-08T01:01:14.064 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-modules-5.15.0-106-generic.
2024-05-08T01:01:16.009 INFO:teuthology.orchestra.run.smithi142.stdout:(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 129542 files and directories currently installed.)
2024-05-08T01:01:16.013 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../0-linux-modules-5.15.0-106-generic_5.15.0-106.116_amd64.deb ...
2024-05-08T01:01:16.237 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-modules-5.15.0-106-generic (5.15.0-106.116) ...
2024-05-08T01:01:18.585 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-image-5.15.0-106-generic.
2024-05-08T01:01:18.606 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../1-linux-image-5.15.0-106-generic_5.15.0-106.116_amd64.deb ...
2024-05-08T01:01:18.683 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-image-5.15.0-106-generic (5.15.0-106.116) ...
2024-05-08T01:01:19.213 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-106-generic.
2024-05-08T01:01:19.233 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-106-generic_5.15.0-106.116_amd64.deb ...
2024-05-08T01:01:19.290 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-modules-extra-5.15.0-106-generic (5.15.0-106.116) ...
2024-05-08T01:01:26.783 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../3-linux-generic_5.15.0.106.106_amd64.deb ...
2024-05-08T01:01:26.918 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-generic (5.15.0.106.106) over (5.15.0.56.54) ...
2024-05-08T01:01:27.268 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.106.106_amd64.deb ...
2024-05-08T01:01:27.388 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-image-generic (5.15.0.106.106) over (5.15.0.56.54) ...
2024-05-08T01:01:27.766 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-headers-5.15.0-106.
2024-05-08T01:01:27.788 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../5-linux-headers-5.15.0-106_5.15.0-106.116_all.deb ...
2024-05-08T01:01:27.832 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-headers-5.15.0-106 (5.15.0-106.116) ...
2024-05-08T01:01:31.038 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.106
2024-05-08T01:01:33.836 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-headers-5.15.0-106-generic.
2024-05-08T01:01:33.873 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../6-linux-headers-5.15.0-106-generic_5.15.0-106.116_amd64.deb ...
2024-05-08T01:01:33.936 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-headers-5.15.0-106-generic (5.15.0-106.116) ...
2024-05-08T01:01:37.386 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.106.106_amd64.deb ...
2024-05-08T01:01:37.528 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-headers-generic (5.15.0.106.106) over (5.15.0.56.54) ...
2024-05-08T01:01:37.919 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-headers-5.15.0-106 (5.15.0-106.116) ...
2024-05-08T01:01:38.074 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-headers-5.15.0-106-generic (5.15.0-106.116) ...
2024-05-08T01:01:38.183 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-headers-generic (5.15.0.106.106) ...
2024-05-08T01:01:38.311 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-image-5.15.0-106-generic (5.15.0-106.116) ...
2024-05-08T01:01:39.638 INFO:teuthology.orchestra.run.smithi142.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic
2024-05-08T01:01:39.638 INFO:teuthology.orchestra.run.smithi142.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic
2024-05-08T01:01:39.638 INFO:teuthology.orchestra.run.smithi142.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-106-generic
2024-05-08T01:01:39.638 INFO:teuthology.orchestra.run.smithi142.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-106-generic
2024-05-08T01:01:39.829 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-modules-5.15.0-106-generic (5.15.0-106.116) ...
2024-05-08T01:01:40.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com'
2024-05-08T01:01:40.040 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T01:01:41.255 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-modules-extra-5.15.0-106-generic (5.15.0-106.116) ...
2024-05-08T01:01:42.682 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-image-generic (5.15.0.106.106) ...
2024-05-08T01:01:42.832 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-generic (5.15.0.106.106) ...
2024-05-08T01:01:42.942 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for linux-image-5.15.0-106-generic (5.15.0-106.116) ...
2024-05-08T01:01:43.001 INFO:teuthology.orchestra.run.smithi142.stdout:/etc/kernel/postinst.d/initramfs-tools:
2024-05-08T01:01:43.001 INFO:teuthology.orchestra.run.smithi142.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-106-generic
2024-05-08T01:01:43.102 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.106
2024-05-08T01:01:55.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com'
2024-05-08T01:01:55.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T01:01:57.170 INFO:teuthology.orchestra.run.smithi142.stdout:/etc/kernel/postinst.d/zz-update-grub:
2024-05-08T01:01:57.171 INFO:teuthology.orchestra.run.smithi142.stdout:Sourcing file `/etc/default/grub'
2024-05-08T01:01:57.196 INFO:teuthology.orchestra.run.smithi142.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg'
2024-05-08T01:01:57.219 INFO:teuthology.orchestra.run.smithi142.stdout:Generating grub configuration file ...
2024-05-08T01:01:57.556 INFO:teuthology.orchestra.run.smithi142.stdout:Found linux image: /boot/vmlinuz-5.15.0-106-generic
2024-05-08T01:01:57.582 INFO:teuthology.orchestra.run.smithi142.stdout:Found initrd image: /boot/initrd.img-5.15.0-106-generic
2024-05-08T01:01:57.908 INFO:teuthology.orchestra.run.smithi142.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic
2024-05-08T01:01:57.914 INFO:teuthology.orchestra.run.smithi142.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic
2024-05-08T01:01:57.988 INFO:teuthology.orchestra.run.smithi142.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic
2024-05-08T01:01:57.993 INFO:teuthology.orchestra.run.smithi142.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic
2024-05-08T01:01:58.196 INFO:teuthology.orchestra.run.smithi142.stdout:Warning: os-prober will not be executed to detect other bootable partitions.
2024-05-08T01:01:58.196 INFO:teuthology.orchestra.run.smithi142.stdout:Systems on them will not be added to the GRUB boot configuration.
2024-05-08T01:01:58.196 INFO:teuthology.orchestra.run.smithi142.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry.
2024-05-08T01:01:58.246 INFO:teuthology.orchestra.run.smithi142.stdout:done
2024-05-08T01:01:58.715 DEBUG:teuthology.orchestra.run.smithi142:> dpkg -s linux-image-generic
2024-05-08T01:01:58.735 INFO:teuthology.orchestra.run.smithi142.stdout:Package: linux-image-generic
2024-05-08T01:01:58.735 INFO:teuthology.orchestra.run.smithi142.stdout:Status: install ok installed
2024-05-08T01:01:58.736 INFO:teuthology.orchestra.run.smithi142.stdout:Priority: optional
2024-05-08T01:01:58.736 INFO:teuthology.orchestra.run.smithi142.stdout:Section: kernel
2024-05-08T01:01:58.736 INFO:teuthology.orchestra.run.smithi142.stdout:Installed-Size: 21
2024-05-08T01:01:58.736 INFO:teuthology.orchestra.run.smithi142.stdout:Maintainer: Ubuntu Kernel Team <kernel-team@lists.ubuntu.com>
2024-05-08T01:01:58.736 INFO:teuthology.orchestra.run.smithi142.stdout:Architecture: amd64
2024-05-08T01:01:58.736 INFO:teuthology.orchestra.run.smithi142.stdout:Source: linux-meta
2024-05-08T01:01:58.736 INFO:teuthology.orchestra.run.smithi142.stdout:Version: 5.15.0.106.106
2024-05-08T01:01:58.736 INFO:teuthology.orchestra.run.smithi142.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-106), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3)
2024-05-08T01:01:58.736 INFO:teuthology.orchestra.run.smithi142.stdout:Depends: linux-image-5.15.0-106-generic, linux-modules-extra-5.15.0-106-generic, linux-firmware, intel-microcode, amd64-microcode
2024-05-08T01:01:58.736 INFO:teuthology.orchestra.run.smithi142.stdout:Recommends: thermald
2024-05-08T01:01:58.736 INFO:teuthology.orchestra.run.smithi142.stdout:Description: Generic Linux kernel image
2024-05-08T01:01:58.736 INFO:teuthology.orchestra.run.smithi142.stdout: This package will always depend on the latest generic kernel image
2024-05-08T01:01:58.736 INFO:teuthology.orchestra.run.smithi142.stdout: available.
2024-05-08T01:01:58.737 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-106-generic
2024-05-08T01:01:58.737 INFO:teuthology.task.kernel:Skipping firmware on distro kernel
2024-05-08T01:01:58.737 DEBUG:teuthology.task.kernel:src is distro, skipping download
2024-05-08T01:01:58.737 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a...
2024-05-08T01:01:58.737 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi142.front.sepia.ceph.com, path=None, version=distro)
2024-05-08T01:01:58.737 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get clean
2024-05-08T01:01:58.906 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get update
2024-05-08T01:01:59.023 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease
2024-05-08T01:01:59.039 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease
2024-05-08T01:01:59.060 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
2024-05-08T01:01:59.081 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
2024-05-08T01:02:00.418 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists...
2024-05-08T01:02:00.437 DEBUG:teuthology.orchestra.run.smithi142:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic
2024-05-08T01:02:00.488 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists...
2024-05-08T01:02:00.672 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree...
2024-05-08T01:02:00.673 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information...
2024-05-08T01:02:00.831 INFO:teuthology.orchestra.run.smithi142.stdout:linux-image-generic is already the newest version (5.15.0.106.106).
2024-05-08T01:02:00.831 INFO:teuthology.orchestra.run.smithi142.stdout:The following packages were automatically installed and are no longer required:
2024-05-08T01:02:00.831 INFO:teuthology.orchestra.run.smithi142.stdout:  libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0
2024-05-08T01:02:00.832 INFO:teuthology.orchestra.run.smithi142.stdout:  libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib
2024-05-08T01:02:00.832 INFO:teuthology.orchestra.run.smithi142.stdout:  libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib
2024-05-08T01:02:00.832 INFO:teuthology.orchestra.run.smithi142.stdout:  linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2
2024-05-08T01:02:00.832 INFO:teuthology.orchestra.run.smithi142.stdout:  python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal
2024-05-08T01:02:00.832 INFO:teuthology.orchestra.run.smithi142.stdout:Use 'sudo apt autoremove' to remove them.
2024-05-08T01:02:00.870 INFO:teuthology.orchestra.run.smithi142.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded.
2024-05-08T01:02:00.872 DEBUG:teuthology.orchestra.run.smithi142:> dpkg -s linux-image-generic
2024-05-08T01:02:00.925 INFO:teuthology.orchestra.run.smithi142.stdout:Package: linux-image-generic
2024-05-08T01:02:00.925 INFO:teuthology.orchestra.run.smithi142.stdout:Status: install ok installed
2024-05-08T01:02:00.925 INFO:teuthology.orchestra.run.smithi142.stdout:Priority: optional
2024-05-08T01:02:00.925 INFO:teuthology.orchestra.run.smithi142.stdout:Section: kernel
2024-05-08T01:02:00.925 INFO:teuthology.orchestra.run.smithi142.stdout:Installed-Size: 21
2024-05-08T01:02:00.925 INFO:teuthology.orchestra.run.smithi142.stdout:Maintainer: Ubuntu Kernel Team <kernel-team@lists.ubuntu.com>
2024-05-08T01:02:00.925 INFO:teuthology.orchestra.run.smithi142.stdout:Architecture: amd64
2024-05-08T01:02:00.925 INFO:teuthology.orchestra.run.smithi142.stdout:Source: linux-meta
2024-05-08T01:02:00.926 INFO:teuthology.orchestra.run.smithi142.stdout:Version: 5.15.0.106.106
2024-05-08T01:02:00.926 INFO:teuthology.orchestra.run.smithi142.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-106), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3)
2024-05-08T01:02:00.926 INFO:teuthology.orchestra.run.smithi142.stdout:Depends: linux-image-5.15.0-106-generic, linux-modules-extra-5.15.0-106-generic, linux-firmware, intel-microcode, amd64-microcode
2024-05-08T01:02:00.926 INFO:teuthology.orchestra.run.smithi142.stdout:Recommends: thermald
2024-05-08T01:02:00.926 INFO:teuthology.orchestra.run.smithi142.stdout:Description: Generic Linux kernel image
2024-05-08T01:02:00.926 INFO:teuthology.orchestra.run.smithi142.stdout: This package will always depend on the latest generic kernel image
2024-05-08T01:02:00.926 INFO:teuthology.orchestra.run.smithi142.stdout: available.
2024-05-08T01:02:00.926 DEBUG:teuthology.orchestra.run.smithi142:> mktemp
2024-05-08T01:02:00.969 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.ryunRyW7FX
2024-05-08T01:02:00.969 DEBUG:teuthology.orchestra.run.smithi142:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ryunRyW7FX
2024-05-08T01:02:01.031 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.ryunRyW7FX
2024-05-08T01:02:01.148 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.ryunRyW7FX is 10KB
2024-05-08T01:02:01.161 DEBUG:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.ryunRyW7FX
2024-05-08T01:02:01.165 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/grub.d/01_ceph_kernel
2024-05-08T01:02:01.218 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:02:01.219 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/grub.d/01_ceph_kernel
2024-05-08T01:02:01.219 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 755 /etc/grub.d/01_ceph_kernel
2024-05-08T01:02:01.282 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-106-generic
2024-05-08T01:02:01.282 DEBUG:teuthology.orchestra.run.smithi142:> sudo update-grub
2024-05-08T01:02:02.800 INFO:teuthology.orchestra.run.smithi142.stderr:Sourcing file `/etc/default/grub'
2024-05-08T01:02:02.801 INFO:teuthology.orchestra.run.smithi142.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg'
2024-05-08T01:02:02.812 INFO:teuthology.orchestra.run.smithi142.stderr:Generating grub configuration file ...
2024-05-08T01:02:03.026 INFO:teuthology.orchestra.run.smithi142.stderr:Found linux image: /boot/vmlinuz-5.15.0-106-generic
2024-05-08T01:02:03.032 INFO:teuthology.orchestra.run.smithi142.stderr:Found initrd image: /boot/initrd.img-5.15.0-106-generic
2024-05-08T01:02:03.269 INFO:teuthology.orchestra.run.smithi142.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic
2024-05-08T01:02:03.273 INFO:teuthology.orchestra.run.smithi142.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic
2024-05-08T01:02:03.314 INFO:teuthology.orchestra.run.smithi142.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic
2024-05-08T01:02:03.318 INFO:teuthology.orchestra.run.smithi142.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic
2024-05-08T01:02:03.431 INFO:teuthology.orchestra.run.smithi142.stderr:Warning: os-prober will not be executed to detect other bootable partitions.
2024-05-08T01:02:03.431 INFO:teuthology.orchestra.run.smithi142.stderr:Systems on them will not be added to the GRUB boot configuration.
2024-05-08T01:02:03.431 INFO:teuthology.orchestra.run.smithi142.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry.
2024-05-08T01:02:03.442 INFO:teuthology.orchestra.run.smithi142.stderr:done
2024-05-08T01:02:03.454 DEBUG:teuthology.orchestra.run.smithi142:> sudo shutdown -r now
2024-05-08T01:02:13.534 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.106
2024-05-08T01:02:28.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com'
2024-05-08T01:02:28.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T01:02:28.760 DEBUG:teuthology.orchestra.run.smithi106:> true
2024-05-08T01:02:29.382 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi106.front.sepia.ceph.com'
2024-05-08T01:02:29.382 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.15.0-106-generic"...
2024-05-08T01:02:29.383 DEBUG:teuthology.orchestra.run.smithi106:> uname -r
2024-05-08T01:02:29.430 INFO:teuthology.orchestra.run.smithi106.stdout:5.15.0-106-generic
2024-05-08T01:02:29.430 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-106-generic vs 5.15.0-106-generic
2024-05-08T01:02:29.430 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install
2024-05-08T01:02:29.430 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu
2024-05-08T01:02:30.431 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a...
2024-05-08T01:02:30.432 DEBUG:teuthology.orchestra.run.smithi106:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc
2024-05-08T01:02:30.774 INFO:teuthology.orchestra.run.smithi106.stdout:ttyS1
2024-05-08T01:02:30.799 DEBUG:teuthology.parallel:result is None
2024-05-08T01:02:33.500 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version...
2024-05-08T01:02:33.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2024-05-08T01:02:33.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T01:02:51.934 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.142
2024-05-08T01:03:00.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2024-05-08T01:03:00.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T01:03:03.998 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.142
2024-05-08T01:03:15.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2024-05-08T01:03:16.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T01:03:47.692 DEBUG:teuthology.orchestra.run.smithi142:> true
2024-05-08T01:03:48.281 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi142.front.sepia.ceph.com'
2024-05-08T01:03:48.281 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.15.0-106-generic"...
2024-05-08T01:03:48.282 DEBUG:teuthology.orchestra.run.smithi142:> uname -r
2024-05-08T01:03:48.327 INFO:teuthology.orchestra.run.smithi142.stdout:5.15.0-106-generic
2024-05-08T01:03:48.328 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-106-generic vs 5.15.0-106-generic
2024-05-08T01:03:48.328 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install
2024-05-08T01:03:48.328 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu
2024-05-08T01:03:49.329 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a...
2024-05-08T01:03:49.329 DEBUG:teuthology.orchestra.run.smithi142:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc
2024-05-08T01:03:49.646 INFO:teuthology.orchestra.run.smithi142.stdout:ttyS1
2024-05-08T01:03:49.673 DEBUG:teuthology.parallel:result is None
2024-05-08T01:03:49.674 INFO:teuthology.run_tasks:Running task internal.base...
2024-05-08T01:03:49.681 INFO:teuthology.task.internal:Creating test directory...
2024-05-08T01:03:49.681 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest
2024-05-08T01:03:49.684 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest
2024-05-08T01:03:49.690 INFO:teuthology.run_tasks:Running task internal.archive_upload...
2024-05-08T01:03:49.697 INFO:teuthology.run_tasks:Running task internal.archive...
2024-05-08T01:03:49.706 INFO:teuthology.task.internal:Creating archive directory...
2024-05-08T01:03:49.707 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive
2024-05-08T01:03:49.734 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive
2024-05-08T01:03:49.758 INFO:teuthology.run_tasks:Running task internal.coredump...
2024-05-08T01:03:49.765 INFO:teuthology.task.internal:Enabling coredump saving...
2024-05-08T01:03:49.765 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf
2024-05-08T01:03:49.797 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf
2024-05-08T01:03:49.818 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core
2024-05-08T01:03:49.818 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core
2024-05-08T01:03:49.826 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core
2024-05-08T01:03:49.826 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core
2024-05-08T01:03:49.828 INFO:teuthology.run_tasks:Running task internal.sudo...
2024-05-08T01:03:49.835 INFO:teuthology.task.internal:Configuring sudo...
2024-05-08T01:03:49.835 DEBUG:teuthology.orchestra.run.smithi106:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers
2024-05-08T01:03:49.873 DEBUG:teuthology.orchestra.run.smithi142:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers
2024-05-08T01:03:49.889 INFO:teuthology.run_tasks:Running task internal.syslog...
2024-05-08T01:03:49.899 INFO:teuthology.task.internal.syslog:Starting syslog monitoring...
2024-05-08T01:03:49.899 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog
2024-05-08T01:03:49.933 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog
2024-05-08T01:03:49.939 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log
2024-05-08T01:03:49.985 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log
2024-05-08T01:03:50.032 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:03:50.033 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf
2024-05-08T01:03:50.100 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log
2024-05-08T01:03:50.106 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log
2024-05-08T01:03:50.152 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:03:50.152 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf
2024-05-08T01:03:50.218 DEBUG:teuthology.orchestra.run.smithi106:> sudo service rsyslog restart
2024-05-08T01:03:50.221 DEBUG:teuthology.orchestra.run.smithi142:> sudo service rsyslog restart
2024-05-08T01:03:50.293 INFO:teuthology.run_tasks:Running task internal.timer...
2024-05-08T01:03:50.300 INFO:teuthology.task.internal:Starting timer...
2024-05-08T01:03:50.300 INFO:teuthology.run_tasks:Running task pcp...
2024-05-08T01:03:50.314 INFO:teuthology.run_tasks:Running task selinux...
2024-05-08T01:03:50.321 DEBUG:teuthology.task.selinux:Excluding smithi106: OS 'ubuntu' does not support SELinux
2024-05-08T01:03:50.321 DEBUG:teuthology.task.selinux:Excluding smithi142: OS 'ubuntu' does not support SELinux
2024-05-08T01:03:50.322 DEBUG:teuthology.task.selinux:Getting current SELinux state
2024-05-08T01:03:50.322 DEBUG:teuthology.task.selinux:Existing SELinux modes: {}
2024-05-08T01:03:50.322 INFO:teuthology.task.selinux:Putting SELinux into permissive mode
2024-05-08T01:03:50.322 INFO:teuthology.run_tasks:Running task ansible.cephlab...
2024-05-08T01:03:50.332 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main
2024-05-08T01:03:50.349 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}]
2024-05-08T01:03:50.351 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi106.front.sepia.ceph.com,smithi142.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml
2024-05-08T01:10:50.402 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi106.front.sepia.ceph.com'), Remote(name='ubuntu@smithi142.front.sepia.ceph.com')]
2024-05-08T01:10:50.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com'
2024-05-08T01:10:50.404 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T01:10:50.487 DEBUG:teuthology.orchestra.run.smithi106:> true
2024-05-08T01:10:50.566 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi106.front.sepia.ceph.com'
2024-05-08T01:10:50.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2024-05-08T01:10:50.567 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T01:10:50.649 DEBUG:teuthology.orchestra.run.smithi142:> true
2024-05-08T01:10:50.734 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi142.front.sepia.ceph.com'
2024-05-08T01:10:50.734 INFO:teuthology.run_tasks:Running task clock...
2024-05-08T01:10:50.746 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew...
2024-05-08T01:10:50.746 INFO:teuthology.orchestra.run:Running command with timeout 360
2024-05-08T01:10:50.746 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true
2024-05-08T01:10:50.749 INFO:teuthology.orchestra.run:Running command with timeout 360
2024-05-08T01:10:50.749 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true
2024-05-08T01:10:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting
2024-05-08T01:10:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: Command line: ntpd -gq
2024-05-08T01:10:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: ----------------------------------------------------
2024-05-08T01:10:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: ntp-4 is maintained by Network Time Foundation,
2024-05-08T01:10:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: Inc. (NTF), a non-profit 501(c)(3) public-benefit
2024-05-08T01:10:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: corporation.  Support and training for ntp-4 are
2024-05-08T01:10:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: available at https://www.nwtime.org/support
2024-05-08T01:10:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: ----------------------------------------------------
2024-05-08T01:10:50.779 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: proto: precision = 0.061 usec (-24)
2024-05-08T01:10:50.779 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: basedate set to 2022-02-04
2024-05-08T01:10:50.779 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: gps base set to 2022-02-06 (week 2196)
2024-05-08T01:10:50.779 INFO:teuthology.orchestra.run.smithi106.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED.
2024-05-08T01:10:50.780 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: restrict 0.0.0.0: KOD does nothing without LIMITED.
2024-05-08T01:10:50.780 INFO:teuthology.orchestra.run.smithi106.stderr:restrict ::: KOD does nothing without LIMITED.
2024-05-08T01:10:50.781 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: restrict ::: KOD does nothing without LIMITED.
2024-05-08T01:10:50.781 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: Listen and drop on 0 v6wildcard [::]:123
2024-05-08T01:10:50.781 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: Listen and drop on 1 v4wildcard 0.0.0.0:123
2024-05-08T01:10:50.781 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: Listen normally on 2 lo 127.0.0.1:123
2024-05-08T01:10:50.781 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: Listen normally on 3 enp3s0f1 172.21.15.106:123
2024-05-08T01:10:50.781 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: Listen normally on 4 lo [::1]:123
2024-05-08T01:10:50.781 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:ce6d%5]:123
2024-05-08T01:10:50.782 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:50 ntpd[17262]: Listening on routing socket on fd #22 for interface updates
2024-05-08T01:10:50.800 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting
2024-05-08T01:10:50.800 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: Command line: ntpd -gq
2024-05-08T01:10:50.801 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: ----------------------------------------------------
2024-05-08T01:10:50.801 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: ntp-4 is maintained by Network Time Foundation,
2024-05-08T01:10:50.801 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: Inc. (NTF), a non-profit 501(c)(3) public-benefit
2024-05-08T01:10:50.801 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: corporation.  Support and training for ntp-4 are
2024-05-08T01:10:50.801 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: available at https://www.nwtime.org/support
2024-05-08T01:10:50.801 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: ----------------------------------------------------
2024-05-08T01:10:50.802 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: proto: precision = 0.046 usec (-24)
2024-05-08T01:10:50.802 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: basedate set to 2022-02-04
2024-05-08T01:10:50.802 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: gps base set to 2022-02-06 (week 2196)
2024-05-08T01:10:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED.
2024-05-08T01:10:50.802 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: restrict 0.0.0.0: KOD does nothing without LIMITED.
2024-05-08T01:10:50.802 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: restrict ::: KOD does nothing without LIMITED.
2024-05-08T01:10:50.803 INFO:teuthology.orchestra.run.smithi142.stderr:restrict ::: KOD does nothing without LIMITED.
2024-05-08T01:10:50.804 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: Listen and drop on 0 v6wildcard [::]:123
2024-05-08T01:10:50.804 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: Listen and drop on 1 v4wildcard 0.0.0.0:123
2024-05-08T01:10:50.804 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: Listen normally on 2 lo 127.0.0.1:123
2024-05-08T01:10:50.804 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: Listen normally on 3 enp3s0f1 172.21.15.142:123
2024-05-08T01:10:50.804 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: Listen normally on 4 lo [::1]:123
2024-05-08T01:10:50.804 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6e0d%5]:123
2024-05-08T01:10:50.804 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:50 ntpd[17267]: Listening on routing socket on fd #22 for interface updates
2024-05-08T01:10:51.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:51 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:51.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:51 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:51.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:51 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:51.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:51 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:51.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:51 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:51.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:51 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:51.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:51 ntpd[17267]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2024-05-08T01:10:51.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:51 ntpd[17267]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied
2024-05-08T01:10:51.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:51 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:51.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:51 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:51.803 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:51 ntpd[17267]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2024-05-08T01:10:51.803 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:51 ntpd[17267]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied
2024-05-08T01:10:52.778 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:52 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:52.778 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:52 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:53.779 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:53 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:53.779 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:53 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:53.779 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:53 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:53.779 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:53 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:53.779 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:53 ntpd[17262]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2024-05-08T01:10:53.779 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:53 ntpd[17262]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied
2024-05-08T01:10:53.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:53 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:53.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:53 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:53.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:53 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:53.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:53 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:53.803 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:53 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:53.803 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:53 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:53.803 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:53 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:53.803 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:53 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:53.803 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:53 ntpd[17267]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2024-05-08T01:10:53.803 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:53 ntpd[17267]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied
2024-05-08T01:10:53.803 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:53 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:53.803 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:53 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:53.803 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:53 ntpd[17267]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2024-05-08T01:10:53.803 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:53 ntpd[17267]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied
2024-05-08T01:10:54.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:54 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:54.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:54 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:54.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:54 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:54.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:54 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:54.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:54 ntpd[17267]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2024-05-08T01:10:54.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:54 ntpd[17267]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied
2024-05-08T01:10:55.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:55 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:55.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:55 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:55.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:55 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:55.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:55 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:55.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:55 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:55.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:55 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:55.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:55 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:55.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:55 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:55.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:55 ntpd[17262]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2024-05-08T01:10:55.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:55 ntpd[17262]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied
2024-05-08T01:10:55.781 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:55 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:55.781 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:55 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:55.781 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:55 ntpd[17262]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2024-05-08T01:10:55.781 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:55 ntpd[17262]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied
2024-05-08T01:10:55.781 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:55 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:55.781 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:55 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:55.781 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:55 ntpd[17262]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2024-05-08T01:10:55.781 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:55 ntpd[17262]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied
2024-05-08T01:10:55.801 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:55 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:55.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:55 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:55.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:55 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:55.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:55 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:55.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:55 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:55.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:55 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:55.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:55 ntpd[17267]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2024-05-08T01:10:55.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:55 ntpd[17267]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied
2024-05-08T01:10:55.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:55 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:55.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:55 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:55.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:55 ntpd[17267]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2024-05-08T01:10:55.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:55 ntpd[17267]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied
2024-05-08T01:10:56.801 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:56 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:56.801 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:56 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:56.801 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:56 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:56.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:56 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:56.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:56 ntpd[17267]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2024-05-08T01:10:56.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:56 ntpd[17267]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied
2024-05-08T01:10:57.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:57 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:57.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:57 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:57.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:57 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:57.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:57 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:57.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:57 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:57.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:57 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:57.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:57 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:57.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:57 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:57.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:57 ntpd[17262]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2024-05-08T01:10:57.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:57 ntpd[17262]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied
2024-05-08T01:10:57.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:57 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:57.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:57 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:57.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:57 ntpd[17262]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2024-05-08T01:10:57.781 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:57 ntpd[17262]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied
2024-05-08T01:10:57.781 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:57 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:57.781 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:57 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:57.781 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:57 ntpd[17262]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2024-05-08T01:10:57.781 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:57 ntpd[17262]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied
2024-05-08T01:10:57.801 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:57 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:57.801 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:57 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:57.801 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:57 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:57.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:57 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:57.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:57 ntpd[17267]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:57.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:57 ntpd[17267]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:57.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:57 ntpd[17267]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2024-05-08T01:10:57.802 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:57 ntpd[17267]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied
2024-05-08T01:10:57.802 INFO:teuthology.orchestra.run.smithi142.stdout: 8 May 01:10:57 ntpd[17267]: ntpd: time slew -0.001789 s
2024-05-08T01:10:57.802 INFO:teuthology.orchestra.run.smithi142.stdout:ntpd: time slew -0.001789s
2024-05-08T01:10:57.803 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:57 ntpd[17267]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied
2024-05-08T01:10:57.803 INFO:teuthology.orchestra.run.smithi142.stderr: 8 May 01:10:57 ntpd[17267]: can't open /var/log/ntpstats/loopstats.20240508: Permission denied
2024-05-08T01:10:57.856 INFO:teuthology.orchestra.run.smithi142.stdout:     remote           refid      st t when poll reach   delay   offset  jitter
2024-05-08T01:10:57.856 INFO:teuthology.orchestra.run.smithi142.stdout:==============================================================================
2024-05-08T01:10:57.856 INFO:teuthology.orchestra.run.smithi142.stdout: hv01.front.sepi .INIT.          16 u    -   64    0    0.000   +0.000   0.000
2024-05-08T01:10:57.856 INFO:teuthology.orchestra.run.smithi142.stdout: hv02.front.sepi .INIT.          16 u    -   64    0    0.000   +0.000   0.000
2024-05-08T01:10:57.856 INFO:teuthology.orchestra.run.smithi142.stdout: hv03.front.sepi .INIT.          16 u    -   64    0    0.000   +0.000   0.000
2024-05-08T01:10:57.856 INFO:teuthology.orchestra.run.smithi142.stdout: hv04.front.sepi .INIT.          16 u    -   64    0    0.000   +0.000   0.000
2024-05-08T01:10:59.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:59 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:59.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:59 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:59.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:59 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:59.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:59 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:59.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:59 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:59.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:59 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:59.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:59 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:59.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:59 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:59.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:59 ntpd[17262]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2024-05-08T01:10:59.780 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:59 ntpd[17262]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied
2024-05-08T01:10:59.781 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:59 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2024-05-08T01:10:59.781 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:59 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied
2024-05-08T01:10:59.781 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:59 ntpd[17262]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2024-05-08T01:10:59.781 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:59 ntpd[17262]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied
2024-05-08T01:10:59.781 INFO:teuthology.orchestra.run.smithi106.stdout: 8 May 01:10:59 ntpd[17262]: ntpd: time slew -0.001633 s
2024-05-08T01:10:59.781 INFO:teuthology.orchestra.run.smithi106.stdout:ntpd: time slew -0.001633s
2024-05-08T01:10:59.782 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:59 ntpd[17262]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied
2024-05-08T01:10:59.782 INFO:teuthology.orchestra.run.smithi106.stderr: 8 May 01:10:59 ntpd[17262]: can't open /var/log/ntpstats/loopstats.20240508: Permission denied
2024-05-08T01:10:59.834 INFO:teuthology.orchestra.run.smithi106.stdout:     remote           refid      st t when poll reach   delay   offset  jitter
2024-05-08T01:10:59.834 INFO:teuthology.orchestra.run.smithi106.stdout:==============================================================================
2024-05-08T01:10:59.834 INFO:teuthology.orchestra.run.smithi106.stdout: hv01.front.sepi .INIT.          16 u    -   64    0    0.000   +0.000   0.000
2024-05-08T01:10:59.834 INFO:teuthology.orchestra.run.smithi106.stdout: hv02.front.sepi .INIT.          16 u    -   64    0    0.000   +0.000   0.000
2024-05-08T01:10:59.834 INFO:teuthology.orchestra.run.smithi106.stdout: hv03.front.sepi .INIT.          16 u    -   64    0    0.000   +0.000   0.000
2024-05-08T01:10:59.834 INFO:teuthology.orchestra.run.smithi106.stdout: hv04.front.sepi .INIT.          16 u    -   64    0    0.000   +0.000   0.000
2024-05-08T01:10:59.835 INFO:teuthology.run_tasks:Running task install...
2024-05-08T01:10:59.843 DEBUG:teuthology.task.install:project ceph
2024-05-08T01:10:59.844 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}}
2024-05-08T01:10:59.844 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}
2024-05-08T01:10:59.844 INFO:teuthology.task.install:Using flavor: default
2024-05-08T01:10:59.851 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']}
2024-05-08T01:10:59.851 INFO:teuthology.task.install:extra packages: ['rbd-mirror']
2024-05-08T01:10:59.851 DEBUG:teuthology.orchestra.run.smithi106:> sudo apt-key list | grep Ceph
2024-05-08T01:10:59.852 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-key list | grep Ceph
2024-05-08T01:10:59.914 INFO:teuthology.orchestra.run.smithi142.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
2024-05-08T01:10:59.933 INFO:teuthology.orchestra.run.smithi106.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
2024-05-08T01:10:59.942 INFO:teuthology.orchestra.run.smithi142.stdout:uid           [ unknown] Ceph automated package build (Ceph automated package build) <sage@newdream.net>
2024-05-08T01:10:59.942 INFO:teuthology.orchestra.run.smithi142.stdout:uid           [ unknown] Ceph.com (release key) <security@ceph.com>
2024-05-08T01:10:59.942 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64
2024-05-08T01:10:59.943 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e0c1691aade07c0b78ccff9531f65d3c80e3e220
2024-05-08T01:10:59.961 INFO:teuthology.orchestra.run.smithi106.stdout:uid           [ unknown] Ceph automated package build (Ceph automated package build) <sage@newdream.net>
2024-05-08T01:10:59.961 INFO:teuthology.orchestra.run.smithi106.stdout:uid           [ unknown] Ceph.com (release key) <security@ceph.com>
2024-05-08T01:10:59.962 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64
2024-05-08T01:10:59.962 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e0c1691aade07c0b78ccff9531f65d3c80e3e220
2024-05-08T01:11:00.161 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default/
2024-05-08T01:11:00.162 INFO:teuthology.task.install.deb:Package version is 19.0.0-2529-ge0c1691a-1jammy
2024-05-08T01:11:00.186 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default/
2024-05-08T01:11:00.187 INFO:teuthology.task.install.deb:Package version is 19.0.0-2529-ge0c1691a-1jammy
2024-05-08T01:11:00.275 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:11:00.275 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/apt/sources.list.d/ceph.list
2024-05-08T01:11:00.288 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get update
2024-05-08T01:11:00.293 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:11:00.293 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/apt/sources.list.d/ceph.list
2024-05-08T01:11:00.307 DEBUG:teuthology.orchestra.run.smithi106:> sudo apt-get update
2024-05-08T01:11:00.483 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease
2024-05-08T01:11:00.484 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
2024-05-08T01:11:00.503 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
2024-05-08T01:11:00.510 INFO:teuthology.orchestra.run.smithi142.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy InRelease
2024-05-08T01:11:00.526 INFO:teuthology.orchestra.run.smithi106.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy InRelease
2024-05-08T01:11:00.532 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy Release [6,978 B]
2024-05-08T01:11:00.532 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
2024-05-08T01:11:00.547 INFO:teuthology.orchestra.run.smithi106.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy Release [6,978 B]
2024-05-08T01:11:00.554 INFO:teuthology.orchestra.run.smithi142.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy Release.gpg
2024-05-08T01:11:00.570 INFO:teuthology.orchestra.run.smithi106.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy Release.gpg
2024-05-08T01:11:00.575 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB]
2024-05-08T01:11:00.591 INFO:teuthology.orchestra.run.smithi106.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,408 B]
2024-05-08T01:11:00.602 INFO:teuthology.orchestra.run.smithi142.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,408 B]
2024-05-08T01:11:00.610 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease
2024-05-08T01:11:00.610 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease
2024-05-08T01:11:00.632 INFO:teuthology.orchestra.run.smithi106.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB]
2024-05-08T01:11:00.676 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
2024-05-08T01:11:00.757 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
2024-05-08T01:11:00.845 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 28.3 kB in 0s (74.8 kB/s)
2024-05-08T01:11:00.965 INFO:teuthology.orchestra.run.smithi106.stdout:Fetched 28.3 kB in 0s (59.1 kB/s)
2024-05-08T01:11:01.829 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists...
2024-05-08T01:11:01.845 DEBUG:teuthology.orchestra.run.smithi142:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2529-ge0c1691a-1jammy cephadm=19.0.0-2529-ge0c1691a-1jammy ceph-mds=19.0.0-2529-ge0c1691a-1jammy ceph-mgr=19.0.0-2529-ge0c1691a-1jammy ceph-common=19.0.0-2529-ge0c1691a-1jammy ceph-fuse=19.0.0-2529-ge0c1691a-1jammy ceph-test=19.0.0-2529-ge0c1691a-1jammy ceph-volume=19.0.0-2529-ge0c1691a-1jammy radosgw=19.0.0-2529-ge0c1691a-1jammy python3-rados=19.0.0-2529-ge0c1691a-1jammy python3-rgw=19.0.0-2529-ge0c1691a-1jammy python3-cephfs=19.0.0-2529-ge0c1691a-1jammy python3-rbd=19.0.0-2529-ge0c1691a-1jammy libcephfs2=19.0.0-2529-ge0c1691a-1jammy libcephfs-dev=19.0.0-2529-ge0c1691a-1jammy librados2=19.0.0-2529-ge0c1691a-1jammy librbd1=19.0.0-2529-ge0c1691a-1jammy rbd-fuse=19.0.0-2529-ge0c1691a-1jammy rbd-mirror=19.0.0-2529-ge0c1691a-1jammy
2024-05-08T01:11:01.897 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists...
2024-05-08T01:11:01.946 INFO:teuthology.orchestra.run.smithi106.stdout:Reading package lists...
2024-05-08T01:11:01.961 DEBUG:teuthology.orchestra.run.smithi106:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2529-ge0c1691a-1jammy cephadm=19.0.0-2529-ge0c1691a-1jammy ceph-mds=19.0.0-2529-ge0c1691a-1jammy ceph-mgr=19.0.0-2529-ge0c1691a-1jammy ceph-common=19.0.0-2529-ge0c1691a-1jammy ceph-fuse=19.0.0-2529-ge0c1691a-1jammy ceph-test=19.0.0-2529-ge0c1691a-1jammy ceph-volume=19.0.0-2529-ge0c1691a-1jammy radosgw=19.0.0-2529-ge0c1691a-1jammy python3-rados=19.0.0-2529-ge0c1691a-1jammy python3-rgw=19.0.0-2529-ge0c1691a-1jammy python3-cephfs=19.0.0-2529-ge0c1691a-1jammy python3-rbd=19.0.0-2529-ge0c1691a-1jammy libcephfs2=19.0.0-2529-ge0c1691a-1jammy libcephfs-dev=19.0.0-2529-ge0c1691a-1jammy librados2=19.0.0-2529-ge0c1691a-1jammy librbd1=19.0.0-2529-ge0c1691a-1jammy rbd-fuse=19.0.0-2529-ge0c1691a-1jammy rbd-mirror=19.0.0-2529-ge0c1691a-1jammy
2024-05-08T01:11:02.021 INFO:teuthology.orchestra.run.smithi106.stdout:Reading package lists...
2024-05-08T01:11:02.074 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree...
2024-05-08T01:11:02.074 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information...
2024-05-08T01:11:02.228 INFO:teuthology.orchestra.run.smithi106.stdout:Building dependency tree...
2024-05-08T01:11:02.229 INFO:teuthology.orchestra.run.smithi106.stdout:Reading state information...
2024-05-08T01:11:02.248 INFO:teuthology.orchestra.run.smithi142.stdout:The following packages were automatically installed and are no longer required:
2024-05-08T01:11:02.249 INFO:teuthology.orchestra.run.smithi142.stdout:  libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0
2024-05-08T01:11:02.249 INFO:teuthology.orchestra.run.smithi142.stdout:  libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib
2024-05-08T01:11:02.249 INFO:teuthology.orchestra.run.smithi142.stdout:  libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib
2024-05-08T01:11:02.249 INFO:teuthology.orchestra.run.smithi142.stdout:  linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2
2024-05-08T01:11:02.249 INFO:teuthology.orchestra.run.smithi142.stdout:  python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal
2024-05-08T01:11:02.249 INFO:teuthology.orchestra.run.smithi142.stdout:Use 'sudo apt autoremove' to remove them.
2024-05-08T01:11:02.250 INFO:teuthology.orchestra.run.smithi142.stdout:The following additional packages will be installed:
2024-05-08T01:11:02.251 INFO:teuthology.orchestra.run.smithi142.stdout:  ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local
2024-05-08T01:11:02.251 INFO:teuthology.orchestra.run.smithi142.stdout:  ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq
2024-05-08T01:11:02.251 INFO:teuthology.orchestra.run.smithi142.stdout:  libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5
2024-05-08T01:11:02.251 INFO:teuthology.orchestra.run.smithi142.stdout:  libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1
2024-05-08T01:11:02.251 INFO:teuthology.orchestra.run.smithi142.stdout:  librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0
2024-05-08T01:11:02.251 INFO:teuthology.orchestra.run.smithi142.stdout:  lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config
2024-05-08T01:11:02.251 INFO:teuthology.orchestra.run.smithi142.stdout:  python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl
2024-05-08T01:11:02.251 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-asyncssh python3-babel python3-bcrypt python3-cachetools
2024-05-08T01:11:02.251 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot
2024-05-08T01:11:02.251 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig
2024-05-08T01:11:02.251 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools
2024-05-08T01:11:02.251 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes
2024-05-08T01:11:02.251 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-logutils python3-mako python3-markupsafe python3-natsort
2024-05-08T01:11:02.251 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-openssl python3-paste python3-pastedeploy python3-pastescript
2024-05-08T01:11:02.252 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-pecan python3-pluggy python3-portend python3-prettytable
2024-05-08T01:11:02.252 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules
2024-05-08T01:11:02.252 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-pygments python3-pyinotify python3-pytest python3-repoze.lru
2024-05-08T01:11:02.252 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-requests python3-requests-oauthlib python3-routes python3-rsa
2024-05-08T01:11:02.252 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-simplegeneric python3-simplejson python3-singledispatch
2024-05-08T01:11:02.252 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-sklearn python3-sklearn-lib python3-tempita python3-tempora
2024-05-08T01:11:02.252 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-threadpoolctl python3-toml python3-urllib3 python3-waitress
2024-05-08T01:11:02.252 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-wcwidth python3-webob python3-websocket python3-webtest
2024-05-08T01:11:02.252 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip
2024-05-08T01:11:02.252 INFO:teuthology.orchestra.run.smithi142.stdout:  xmlstarlet zip
2024-05-08T01:11:02.253 INFO:teuthology.orchestra.run.smithi142.stdout:Suggested packages:
2024-05-08T01:11:02.253 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-influxdb readline-doc python-jinja2-doc python3-beaker
2024-05-08T01:11:02.253 INFO:teuthology.orchestra.run.smithi142.stdout:  python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg
2024-05-08T01:11:02.253 INFO:teuthology.orchestra.run.smithi142.stdout:  httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit
2024-05-08T01:11:02.253 INFO:teuthology.orchestra.run.smithi142.stdout:  python-pecan-doc python-psutil-doc subversion python-pygments-doc
2024-05-08T01:11:02.253 INFO:teuthology.orchestra.run.smithi142.stdout:  python-pyinotify-doc python3-socks python-requests-doc python3-dap
2024-05-08T01:11:02.253 INFO:teuthology.orchestra.run.smithi142.stdout:  python-sklearn-doc ipython3 python-waitress-doc python-webob-doc
2024-05-08T01:11:02.253 INFO:teuthology.orchestra.run.smithi142.stdout:  python-webtest-doc python-werkzeug-doc python3-watchdog
2024-05-08T01:11:02.253 INFO:teuthology.orchestra.run.smithi142.stdout:Recommended packages:
2024-05-08T01:11:02.253 INFO:teuthology.orchestra.run.smithi142.stdout:  btrfs-tools
2024-05-08T01:11:02.351 INFO:teuthology.orchestra.run.smithi142.stdout:The following NEW packages will be installed:
2024-05-08T01:11:02.351 INFO:teuthology.orchestra.run.smithi142.stdout:  ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm
2024-05-08T01:11:02.352 INFO:teuthology.orchestra.run.smithi142.stdout:  ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents
2024-05-08T01:11:02.352 INFO:teuthology.orchestra.run.smithi142.stdout:  ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq
2024-05-08T01:11:02.352 INFO:teuthology.orchestra.run.smithi142.stdout:  libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1
2024-05-08T01:11:02.352 INFO:teuthology.orchestra.run.smithi142.stdout:  liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5
2024-05-08T01:11:02.352 INFO:teuthology.orchestra.run.smithi142.stdout:  libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1
2024-05-08T01:11:02.353 INFO:teuthology.orchestra.run.smithi142.stdout:  libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec
2024-05-08T01:11:02.353 INFO:teuthology.orchestra.run.smithi142.stdout:  lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc
2024-05-08T01:11:02.353 INFO:teuthology.orchestra.run.smithi142.stdout:  python-babel-localedata python-pastedeploy-tpl python3-asyncssh
2024-05-08T01:11:02.353 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse
2024-05-08T01:11:02.353 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-ceph-common python3-cephfs python3-certifi python3-cheroot
2024-05-08T01:11:02.353 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig
2024-05-08T01:11:02.353 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools
2024-05-08T01:11:02.353 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes
2024-05-08T01:11:02.354 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-logutils python3-mako python3-markupsafe python3-natsort
2024-05-08T01:11:02.354 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-openssl python3-paste python3-pastedeploy python3-pastescript
2024-05-08T01:11:02.354 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-pecan python3-pluggy python3-portend python3-prettytable
2024-05-08T01:11:02.354 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules
2024-05-08T01:11:02.354 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd
2024-05-08T01:11:02.354 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw
2024-05-08T01:11:02.354 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-routes python3-rsa python3-simplegeneric python3-simplejson
2024-05-08T01:11:02.354 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita
2024-05-08T01:11:02.354 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-tempora python3-threadpoolctl python3-toml python3-urllib3
2024-05-08T01:11:02.354 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-waitress python3-wcwidth python3-webob python3-websocket
2024-05-08T01:11:02.354 INFO:teuthology.orchestra.run.smithi142.stdout:  python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n
2024-05-08T01:11:02.354 INFO:teuthology.orchestra.run.smithi142.stdout:  radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip
2024-05-08T01:11:02.420 INFO:teuthology.orchestra.run.smithi106.stdout:The following packages were automatically installed and are no longer required:
2024-05-08T01:11:02.420 INFO:teuthology.orchestra.run.smithi106.stdout:  libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0
2024-05-08T01:11:02.420 INFO:teuthology.orchestra.run.smithi106.stdout:  libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib
2024-05-08T01:11:02.420 INFO:teuthology.orchestra.run.smithi106.stdout:  libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib
2024-05-08T01:11:02.421 INFO:teuthology.orchestra.run.smithi106.stdout:  linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2
2024-05-08T01:11:02.421 INFO:teuthology.orchestra.run.smithi106.stdout:  python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal
2024-05-08T01:11:02.421 INFO:teuthology.orchestra.run.smithi106.stdout:Use 'sudo apt autoremove' to remove them.
2024-05-08T01:11:02.422 INFO:teuthology.orchestra.run.smithi106.stdout:The following additional packages will be installed:
2024-05-08T01:11:02.422 INFO:teuthology.orchestra.run.smithi106.stdout:  ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local
2024-05-08T01:11:02.423 INFO:teuthology.orchestra.run.smithi106.stdout:  ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq
2024-05-08T01:11:02.423 INFO:teuthology.orchestra.run.smithi106.stdout:  libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5
2024-05-08T01:11:02.424 INFO:teuthology.orchestra.run.smithi106.stdout:  libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1
2024-05-08T01:11:02.424 INFO:teuthology.orchestra.run.smithi106.stdout:  librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0
2024-05-08T01:11:02.424 INFO:teuthology.orchestra.run.smithi106.stdout:  lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config
2024-05-08T01:11:02.424 INFO:teuthology.orchestra.run.smithi106.stdout:  python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl
2024-05-08T01:11:02.424 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-asyncssh python3-babel python3-bcrypt python3-cachetools
2024-05-08T01:11:02.424 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot
2024-05-08T01:11:02.424 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig
2024-05-08T01:11:02.424 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools
2024-05-08T01:11:02.424 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes
2024-05-08T01:11:02.424 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-logutils python3-mako python3-markupsafe python3-natsort
2024-05-08T01:11:02.424 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-openssl python3-paste python3-pastedeploy python3-pastescript
2024-05-08T01:11:02.424 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-pecan python3-pluggy python3-portend python3-prettytable
2024-05-08T01:11:02.424 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules
2024-05-08T01:11:02.424 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-pygments python3-pyinotify python3-pytest python3-repoze.lru
2024-05-08T01:11:02.425 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-requests python3-requests-oauthlib python3-routes python3-rsa
2024-05-08T01:11:02.425 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-simplegeneric python3-simplejson python3-singledispatch
2024-05-08T01:11:02.425 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-sklearn python3-sklearn-lib python3-tempita python3-tempora
2024-05-08T01:11:02.425 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-threadpoolctl python3-toml python3-urllib3 python3-waitress
2024-05-08T01:11:02.425 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-wcwidth python3-webob python3-websocket python3-webtest
2024-05-08T01:11:02.425 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip
2024-05-08T01:11:02.425 INFO:teuthology.orchestra.run.smithi106.stdout:  xmlstarlet zip
2024-05-08T01:11:02.425 INFO:teuthology.orchestra.run.smithi106.stdout:Suggested packages:
2024-05-08T01:11:02.426 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-influxdb readline-doc python-jinja2-doc python3-beaker
2024-05-08T01:11:02.426 INFO:teuthology.orchestra.run.smithi106.stdout:  python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg
2024-05-08T01:11:02.426 INFO:teuthology.orchestra.run.smithi106.stdout:  httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit
2024-05-08T01:11:02.426 INFO:teuthology.orchestra.run.smithi106.stdout:  python-pecan-doc python-psutil-doc subversion python-pygments-doc
2024-05-08T01:11:02.426 INFO:teuthology.orchestra.run.smithi106.stdout:  python-pyinotify-doc python3-socks python-requests-doc python3-dap
2024-05-08T01:11:02.426 INFO:teuthology.orchestra.run.smithi106.stdout:  python-sklearn-doc ipython3 python-waitress-doc python-webob-doc
2024-05-08T01:11:02.426 INFO:teuthology.orchestra.run.smithi106.stdout:  python-webtest-doc python-werkzeug-doc python3-watchdog
2024-05-08T01:11:02.426 INFO:teuthology.orchestra.run.smithi106.stdout:Recommended packages:
2024-05-08T01:11:02.426 INFO:teuthology.orchestra.run.smithi106.stdout:  btrfs-tools
2024-05-08T01:11:02.525 INFO:teuthology.orchestra.run.smithi106.stdout:The following NEW packages will be installed:
2024-05-08T01:11:02.525 INFO:teuthology.orchestra.run.smithi106.stdout:  ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm
2024-05-08T01:11:02.525 INFO:teuthology.orchestra.run.smithi106.stdout:  ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents
2024-05-08T01:11:02.525 INFO:teuthology.orchestra.run.smithi106.stdout:  ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq
2024-05-08T01:11:02.525 INFO:teuthology.orchestra.run.smithi106.stdout:  libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1
2024-05-08T01:11:02.525 INFO:teuthology.orchestra.run.smithi106.stdout:  liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5
2024-05-08T01:11:02.525 INFO:teuthology.orchestra.run.smithi106.stdout:  libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1
2024-05-08T01:11:02.527 INFO:teuthology.orchestra.run.smithi106.stdout:  libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec
2024-05-08T01:11:02.527 INFO:teuthology.orchestra.run.smithi106.stdout:  lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc
2024-05-08T01:11:02.527 INFO:teuthology.orchestra.run.smithi106.stdout:  python-babel-localedata python-pastedeploy-tpl python3-asyncssh
2024-05-08T01:11:02.527 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse
2024-05-08T01:11:02.527 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-ceph-common python3-cephfs python3-certifi python3-cheroot
2024-05-08T01:11:02.527 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig
2024-05-08T01:11:02.527 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools
2024-05-08T01:11:02.527 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes
2024-05-08T01:11:02.527 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-logutils python3-mako python3-markupsafe python3-natsort
2024-05-08T01:11:02.527 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-openssl python3-paste python3-pastedeploy python3-pastescript
2024-05-08T01:11:02.527 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-pecan python3-pluggy python3-portend python3-prettytable
2024-05-08T01:11:02.527 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules
2024-05-08T01:11:02.528 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd
2024-05-08T01:11:02.528 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw
2024-05-08T01:11:02.528 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-routes python3-rsa python3-simplegeneric python3-simplejson
2024-05-08T01:11:02.528 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita
2024-05-08T01:11:02.528 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-tempora python3-threadpoolctl python3-toml python3-urllib3
2024-05-08T01:11:02.528 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-waitress python3-wcwidth python3-webob python3-websocket
2024-05-08T01:11:02.528 INFO:teuthology.orchestra.run.smithi106.stdout:  python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n
2024-05-08T01:11:02.528 INFO:teuthology.orchestra.run.smithi106.stdout:  radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip
2024-05-08T01:11:02.557 INFO:teuthology.orchestra.run.smithi142.stdout:0 upgraded, 119 newly installed, 0 to remove and 303 not upgraded.
2024-05-08T01:11:02.557 INFO:teuthology.orchestra.run.smithi142.stdout:Need to get 183 MB of archives.
2024-05-08T01:11:02.557 INFO:teuthology.orchestra.run.smithi142.stdout:After this operation, 834 MB of additional disk space will be used.
2024-05-08T01:11:02.557 INFO:teuthology.orchestra.run.smithi142.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB]
2024-05-08T01:11:02.578 INFO:teuthology.orchestra.run.smithi142.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2529-ge0c1691a-1jammy [3,546 kB]
2024-05-08T01:11:02.595 INFO:teuthology.orchestra.run.smithi106.stdout:0 upgraded, 119 newly installed, 0 to remove and 303 not upgraded.
2024-05-08T01:11:02.596 INFO:teuthology.orchestra.run.smithi106.stdout:Need to get 183 MB of archives.
2024-05-08T01:11:02.596 INFO:teuthology.orchestra.run.smithi106.stdout:After this operation, 834 MB of additional disk space will be used.
2024-05-08T01:11:02.596 INFO:teuthology.orchestra.run.smithi106.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB]
2024-05-08T01:11:02.716 INFO:teuthology.orchestra.run.smithi106.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB]
2024-05-08T01:11:02.719 INFO:teuthology.orchestra.run.smithi106.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB]
2024-05-08T01:11:02.748 INFO:teuthology.orchestra.run.smithi106.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB]
2024-05-08T01:11:02.748 INFO:teuthology.orchestra.run.smithi106.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2529-ge0c1691a-1jammy [3,546 kB]
2024-05-08T01:11:02.854 INFO:teuthology.orchestra.run.smithi106.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB]
2024-05-08T01:11:02.860 INFO:teuthology.orchestra.run.smithi106.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB]
2024-05-08T01:11:02.878 INFO:teuthology.orchestra.run.smithi106.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB]
2024-05-08T01:11:02.883 INFO:teuthology.orchestra.run.smithi106.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB]
2024-05-08T01:11:02.884 INFO:teuthology.orchestra.run.smithi106.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB]
2024-05-08T01:11:02.885 INFO:teuthology.orchestra.run.smithi106.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB]
2024-05-08T01:11:02.895 INFO:teuthology.orchestra.run.smithi106.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB]
2024-05-08T01:11:02.898 INFO:teuthology.orchestra.run.smithi106.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB]
2024-05-08T01:11:02.901 INFO:teuthology.orchestra.run.smithi106.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB]
2024-05-08T01:11:02.902 INFO:teuthology.orchestra.run.smithi106.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B]
2024-05-08T01:11:02.919 INFO:teuthology.orchestra.run.smithi106.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB]
2024-05-08T01:11:02.921 INFO:teuthology.orchestra.run.smithi106.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB]
2024-05-08T01:11:02.924 INFO:teuthology.orchestra.run.smithi106.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB]
2024-05-08T01:11:02.926 INFO:teuthology.orchestra.run.smithi106.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB]
2024-05-08T01:11:02.927 INFO:teuthology.orchestra.run.smithi106.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B]
2024-05-08T01:11:02.940 INFO:teuthology.orchestra.run.smithi106.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB]
2024-05-08T01:11:02.941 INFO:teuthology.orchestra.run.smithi106.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B]
2024-05-08T01:11:02.941 INFO:teuthology.orchestra.run.smithi106.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B]
2024-05-08T01:11:02.942 INFO:teuthology.orchestra.run.smithi106.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB]
2024-05-08T01:11:02.942 INFO:teuthology.orchestra.run.smithi106.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB]
2024-05-08T01:11:02.955 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2529-ge0c1691a-1jammy [3,216 kB]
2024-05-08T01:11:02.965 INFO:teuthology.orchestra.run.smithi106.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B]
2024-05-08T01:11:02.965 INFO:teuthology.orchestra.run.smithi106.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B]
2024-05-08T01:11:02.965 INFO:teuthology.orchestra.run.smithi106.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB]
2024-05-08T01:11:02.966 INFO:teuthology.orchestra.run.smithi106.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB]
2024-05-08T01:11:02.966 INFO:teuthology.orchestra.run.smithi106.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB]
2024-05-08T01:11:02.983 INFO:teuthology.orchestra.run.smithi106.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB]
2024-05-08T01:11:02.983 INFO:teuthology.orchestra.run.smithi106.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB]
2024-05-08T01:11:02.984 INFO:teuthology.orchestra.run.smithi106.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB]
2024-05-08T01:11:02.985 INFO:teuthology.orchestra.run.smithi106.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B]
2024-05-08T01:11:02.985 INFO:teuthology.orchestra.run.smithi106.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB]
2024-05-08T01:11:02.986 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB]
2024-05-08T01:11:02.998 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB]
2024-05-08T01:11:03.003 INFO:teuthology.orchestra.run.smithi106.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB]
2024-05-08T01:11:03.004 INFO:teuthology.orchestra.run.smithi106.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB]
2024-05-08T01:11:03.005 INFO:teuthology.orchestra.run.smithi106.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB]
2024-05-08T01:11:03.012 INFO:teuthology.orchestra.run.smithi106.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B]
2024-05-08T01:11:03.012 INFO:teuthology.orchestra.run.smithi106.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB]
2024-05-08T01:11:03.024 INFO:teuthology.orchestra.run.smithi106.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB]
2024-05-08T01:11:03.025 INFO:teuthology.orchestra.run.smithi106.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB]
2024-05-08T01:11:03.026 INFO:teuthology.orchestra.run.smithi106.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB]
2024-05-08T01:11:03.029 INFO:teuthology.orchestra.run.smithi106.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB]
2024-05-08T01:11:03.030 INFO:teuthology.orchestra.run.smithi106.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB]
2024-05-08T01:11:03.045 INFO:teuthology.orchestra.run.smithi106.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB]
2024-05-08T01:11:03.046 INFO:teuthology.orchestra.run.smithi106.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB]
2024-05-08T01:11:03.049 INFO:teuthology.orchestra.run.smithi106.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB]
2024-05-08T01:11:03.049 INFO:teuthology.orchestra.run.smithi106.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB]
2024-05-08T01:11:03.051 INFO:teuthology.orchestra.run.smithi106.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB]
2024-05-08T01:11:03.065 INFO:teuthology.orchestra.run.smithi106.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB]
2024-05-08T01:11:03.071 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2529-ge0c1691a-1jammy [711 kB]
2024-05-08T01:11:03.080 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB]
2024-05-08T01:11:03.111 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2529-ge0c1691a-1jammy [356 kB]
2024-05-08T01:11:03.132 INFO:teuthology.orchestra.run.smithi142.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2529-ge0c1691a-1jammy [32.8 kB]
2024-05-08T01:11:03.133 INFO:teuthology.orchestra.run.smithi142.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2529-ge0c1691a-1jammy [180 kB]
2024-05-08T01:11:03.138 INFO:teuthology.orchestra.run.smithi142.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2529-ge0c1691a-1jammy [67.0 kB]
2024-05-08T01:11:03.142 INFO:teuthology.orchestra.run.smithi142.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2529-ge0c1691a-1jammy [334 kB]
2024-05-08T01:11:03.158 INFO:teuthology.orchestra.run.smithi106.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB]
2024-05-08T01:11:03.159 INFO:teuthology.orchestra.run.smithi106.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB]
2024-05-08T01:11:03.160 INFO:teuthology.orchestra.run.smithi106.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB]
2024-05-08T01:11:03.161 INFO:teuthology.orchestra.run.smithi106.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB]
2024-05-08T01:11:03.162 INFO:teuthology.orchestra.run.smithi106.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB]
2024-05-08T01:11:03.163 INFO:teuthology.orchestra.run.smithi142.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2529-ge0c1691a-1jammy [6,779 kB]
2024-05-08T01:11:03.195 INFO:teuthology.orchestra.run.smithi106.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB]
2024-05-08T01:11:03.199 INFO:teuthology.orchestra.run.smithi106.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB]
2024-05-08T01:11:03.245 INFO:teuthology.orchestra.run.smithi106.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB]
2024-05-08T01:11:03.271 INFO:teuthology.orchestra.run.smithi106.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B]
2024-05-08T01:11:03.271 INFO:teuthology.orchestra.run.smithi106.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB]
2024-05-08T01:11:03.271 INFO:teuthology.orchestra.run.smithi106.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB]
2024-05-08T01:11:03.272 INFO:teuthology.orchestra.run.smithi106.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB]
2024-05-08T01:11:03.272 INFO:teuthology.orchestra.run.smithi106.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB]
2024-05-08T01:11:03.273 INFO:teuthology.orchestra.run.smithi106.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB]
2024-05-08T01:11:03.274 INFO:teuthology.orchestra.run.smithi106.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB]
2024-05-08T01:11:03.274 INFO:teuthology.orchestra.run.smithi106.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB]
2024-05-08T01:11:03.283 INFO:teuthology.orchestra.run.smithi106.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB]
2024-05-08T01:11:03.303 INFO:teuthology.orchestra.run.smithi106.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB]
2024-05-08T01:11:03.305 INFO:teuthology.orchestra.run.smithi106.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB]
2024-05-08T01:11:03.306 INFO:teuthology.orchestra.run.smithi106.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB]
2024-05-08T01:11:03.310 INFO:teuthology.orchestra.run.smithi106.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB]
2024-05-08T01:11:03.311 INFO:teuthology.orchestra.run.smithi142.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB]
2024-05-08T01:11:03.313 INFO:teuthology.orchestra.run.smithi106.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB]
2024-05-08T01:11:03.314 INFO:teuthology.orchestra.run.smithi106.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB]
2024-05-08T01:11:03.315 INFO:teuthology.orchestra.run.smithi106.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB]
2024-05-08T01:11:03.320 INFO:teuthology.orchestra.run.smithi106.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB]
2024-05-08T01:11:03.321 INFO:teuthology.orchestra.run.smithi106.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB]
2024-05-08T01:11:03.323 INFO:teuthology.orchestra.run.smithi142.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB]
2024-05-08T01:11:03.338 INFO:teuthology.orchestra.run.smithi106.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B]
2024-05-08T01:11:03.338 INFO:teuthology.orchestra.run.smithi106.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB]
2024-05-08T01:11:03.339 INFO:teuthology.orchestra.run.smithi106.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB]
2024-05-08T01:11:03.353 INFO:teuthology.orchestra.run.smithi142.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB]
2024-05-08T01:11:03.360 INFO:teuthology.orchestra.run.smithi106.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB]
2024-05-08T01:11:03.362 INFO:teuthology.orchestra.run.smithi142.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB]
2024-05-08T01:11:03.362 INFO:teuthology.orchestra.run.smithi106.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB]
2024-05-08T01:11:03.363 INFO:teuthology.orchestra.run.smithi142.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB]
2024-05-08T01:11:03.364 INFO:teuthology.orchestra.run.smithi106.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB]
2024-05-08T01:11:03.364 INFO:teuthology.orchestra.run.smithi142.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB]
2024-05-08T01:11:03.372 INFO:teuthology.orchestra.run.smithi106.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB]
2024-05-08T01:11:03.372 INFO:teuthology.orchestra.run.smithi106.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB]
2024-05-08T01:11:03.373 INFO:teuthology.orchestra.run.smithi106.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB]
2024-05-08T01:11:03.375 INFO:teuthology.orchestra.run.smithi106.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB]
2024-05-08T01:11:03.383 INFO:teuthology.orchestra.run.smithi142.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB]
2024-05-08T01:11:03.390 INFO:teuthology.orchestra.run.smithi106.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB]
2024-05-08T01:11:03.494 INFO:teuthology.orchestra.run.smithi142.stdout:Get:21 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2529-ge0c1691a-1jammy [112 kB]
2024-05-08T01:11:03.497 INFO:teuthology.orchestra.run.smithi142.stdout:Get:22 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2529-ge0c1691a-1jammy [466 kB]
2024-05-08T01:11:03.509 INFO:teuthology.orchestra.run.smithi142.stdout:Get:23 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2529-ge0c1691a-1jammy [25.6 MB]
2024-05-08T01:11:03.553 INFO:teuthology.orchestra.run.smithi142.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB]
2024-05-08T01:11:03.922 INFO:teuthology.orchestra.run.smithi106.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2529-ge0c1691a-1jammy [3,216 kB]
2024-05-08T01:11:03.965 INFO:teuthology.orchestra.run.smithi142.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB]
2024-05-08T01:11:03.999 INFO:teuthology.orchestra.run.smithi142.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B]
2024-05-08T01:11:04.001 INFO:teuthology.orchestra.run.smithi142.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB]
2024-05-08T01:11:04.083 INFO:teuthology.orchestra.run.smithi142.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB]
2024-05-08T01:11:04.133 INFO:teuthology.orchestra.run.smithi142.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB]
2024-05-08T01:11:04.155 INFO:teuthology.orchestra.run.smithi142.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB]
2024-05-08T01:11:04.162 INFO:teuthology.orchestra.run.smithi142.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B]
2024-05-08T01:11:04.163 INFO:teuthology.orchestra.run.smithi142.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB]
2024-05-08T01:11:04.175 INFO:teuthology.orchestra.run.smithi142.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B]
2024-05-08T01:11:04.175 INFO:teuthology.orchestra.run.smithi142.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B]
2024-05-08T01:11:04.341 INFO:teuthology.orchestra.run.smithi142.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB]
2024-05-08T01:11:04.387 INFO:teuthology.orchestra.run.smithi106.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2529-ge0c1691a-1jammy [711 kB]
2024-05-08T01:11:04.436 INFO:teuthology.orchestra.run.smithi142.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB]
2024-05-08T01:11:04.444 INFO:teuthology.orchestra.run.smithi106.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2529-ge0c1691a-1jammy [356 kB]
2024-05-08T01:11:04.469 INFO:teuthology.orchestra.run.smithi142.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B]
2024-05-08T01:11:04.489 INFO:teuthology.orchestra.run.smithi142.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B]
2024-05-08T01:11:04.497 INFO:teuthology.orchestra.run.smithi106.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2529-ge0c1691a-1jammy [32.8 kB]
2024-05-08T01:11:04.498 INFO:teuthology.orchestra.run.smithi106.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2529-ge0c1691a-1jammy [180 kB]
2024-05-08T01:11:04.502 INFO:teuthology.orchestra.run.smithi106.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2529-ge0c1691a-1jammy [67.0 kB]
2024-05-08T01:11:04.527 INFO:teuthology.orchestra.run.smithi142.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB]
2024-05-08T01:11:04.545 INFO:teuthology.orchestra.run.smithi106.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2529-ge0c1691a-1jammy [334 kB]
2024-05-08T01:11:04.557 INFO:teuthology.orchestra.run.smithi106.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2529-ge0c1691a-1jammy [6,779 kB]
2024-05-08T01:11:04.747 INFO:teuthology.orchestra.run.smithi142.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB]
2024-05-08T01:11:04.762 INFO:teuthology.orchestra.run.smithi142.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB]
2024-05-08T01:11:04.768 INFO:teuthology.orchestra.run.smithi142.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB]
2024-05-08T01:11:04.774 INFO:teuthology.orchestra.run.smithi142.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB]
2024-05-08T01:11:04.795 INFO:teuthology.orchestra.run.smithi142.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB]
2024-05-08T01:11:04.799 INFO:teuthology.orchestra.run.smithi142.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B]
2024-05-08T01:11:04.936 INFO:teuthology.orchestra.run.smithi142.stdout:Get:46 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2529-ge0c1691a-1jammy [5,084 kB]
2024-05-08T01:11:04.964 INFO:teuthology.orchestra.run.smithi142.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB]
2024-05-08T01:11:05.278 INFO:teuthology.orchestra.run.smithi142.stdout:Get:48 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2529-ge0c1691a-1jammy [244 kB]
2024-05-08T01:11:05.295 INFO:teuthology.orchestra.run.smithi142.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB]
2024-05-08T01:11:05.304 INFO:teuthology.orchestra.run.smithi142.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2529-ge0c1691a-1jammy [124 kB]
2024-05-08T01:11:05.308 INFO:teuthology.orchestra.run.smithi142.stdout:Get:51 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2529-ge0c1691a-1jammy [1,526 kB]
2024-05-08T01:11:05.334 INFO:teuthology.orchestra.run.smithi142.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB]
2024-05-08T01:11:05.347 INFO:teuthology.orchestra.run.smithi142.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB]
2024-05-08T01:11:05.433 INFO:teuthology.orchestra.run.smithi142.stdout:Get:54 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2529-ge0c1691a-1jammy [6,246 kB]
2024-05-08T01:11:05.555 INFO:teuthology.orchestra.run.smithi142.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B]
2024-05-08T01:11:05.555 INFO:teuthology.orchestra.run.smithi142.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB]
2024-05-08T01:11:05.561 INFO:teuthology.orchestra.run.smithi142.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB]
2024-05-08T01:11:05.566 INFO:teuthology.orchestra.run.smithi142.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB]
2024-05-08T01:11:05.581 INFO:teuthology.orchestra.run.smithi142.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB]
2024-05-08T01:11:05.602 INFO:teuthology.orchestra.run.smithi142.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB]
2024-05-08T01:11:05.606 INFO:teuthology.orchestra.run.smithi106.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2529-ge0c1691a-1jammy [112 kB]
2024-05-08T01:11:05.610 INFO:teuthology.orchestra.run.smithi142.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB]
2024-05-08T01:11:05.611 INFO:teuthology.orchestra.run.smithi106.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2529-ge0c1691a-1jammy [466 kB]
2024-05-08T01:11:05.684 INFO:teuthology.orchestra.run.smithi106.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2529-ge0c1691a-1jammy [25.6 MB]
2024-05-08T01:11:05.784 INFO:teuthology.orchestra.run.smithi142.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB]
2024-05-08T01:11:05.897 INFO:teuthology.orchestra.run.smithi142.stdout:Get:63 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2529-ge0c1691a-1jammy [22.7 MB]
2024-05-08T01:11:06.040 INFO:teuthology.orchestra.run.smithi142.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB]
2024-05-08T01:11:06.216 INFO:teuthology.orchestra.run.smithi142.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB]
2024-05-08T01:11:06.230 INFO:teuthology.orchestra.run.smithi142.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB]
2024-05-08T01:11:06.250 INFO:teuthology.orchestra.run.smithi142.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB]
2024-05-08T01:11:06.315 INFO:teuthology.orchestra.run.smithi142.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB]
2024-05-08T01:11:06.607 INFO:teuthology.orchestra.run.smithi142.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB]
2024-05-08T01:11:06.609 INFO:teuthology.orchestra.run.smithi142.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB]
2024-05-08T01:11:06.612 INFO:teuthology.orchestra.run.smithi142.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB]
2024-05-08T01:11:06.612 INFO:teuthology.orchestra.run.smithi142.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB]
2024-05-08T01:11:06.614 INFO:teuthology.orchestra.run.smithi142.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB]
2024-05-08T01:11:06.844 INFO:teuthology.orchestra.run.smithi142.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB]
2024-05-08T01:11:07.426 INFO:teuthology.orchestra.run.smithi142.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB]
2024-05-08T01:11:07.442 INFO:teuthology.orchestra.run.smithi142.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB]
2024-05-08T01:11:07.605 INFO:teuthology.orchestra.run.smithi142.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2529-ge0c1691a-1jammy [14.0 kB]
2024-05-08T01:11:07.606 INFO:teuthology.orchestra.run.smithi142.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2529-ge0c1691a-1jammy [869 kB]
2024-05-08T01:11:07.623 INFO:teuthology.orchestra.run.smithi142.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2529-ge0c1691a-1jammy [2,437 kB]
2024-05-08T01:11:07.769 INFO:teuthology.orchestra.run.smithi142.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2529-ge0c1691a-1jammy [791 kB]
2024-05-08T01:11:07.819 INFO:teuthology.orchestra.run.smithi142.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2529-ge0c1691a-1jammy [148 kB]
2024-05-08T01:11:07.843 INFO:teuthology.orchestra.run.smithi142.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2529-ge0c1691a-1jammy [2,331 kB]
2024-05-08T01:11:08.035 INFO:teuthology.orchestra.run.smithi142.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2529-ge0c1691a-1jammy [8,625 kB]
2024-05-08T01:11:08.721 INFO:teuthology.orchestra.run.smithi142.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2529-ge0c1691a-1jammy [14.2 kB]
2024-05-08T01:11:08.721 INFO:teuthology.orchestra.run.smithi142.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2529-ge0c1691a-1jammy [50.7 MB]
2024-05-08T01:11:09.481 INFO:teuthology.orchestra.run.smithi106.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2529-ge0c1691a-1jammy [5,084 kB]
2024-05-08T01:11:10.459 INFO:teuthology.orchestra.run.smithi106.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2529-ge0c1691a-1jammy [244 kB]
2024-05-08T01:11:10.503 INFO:teuthology.orchestra.run.smithi106.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2529-ge0c1691a-1jammy [124 kB]
2024-05-08T01:11:10.509 INFO:teuthology.orchestra.run.smithi106.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2529-ge0c1691a-1jammy [1,526 kB]
2024-05-08T01:11:10.799 INFO:teuthology.orchestra.run.smithi106.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2529-ge0c1691a-1jammy [6,246 kB]
2024-05-08T01:11:12.542 INFO:teuthology.orchestra.run.smithi106.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2529-ge0c1691a-1jammy [22.7 MB]
2024-05-08T01:11:13.437 INFO:teuthology.orchestra.run.smithi142.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B]
2024-05-08T01:11:13.451 INFO:teuthology.orchestra.run.smithi142.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB]
2024-05-08T01:11:13.681 INFO:teuthology.orchestra.run.smithi142.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB]
2024-05-08T01:11:13.961 INFO:teuthology.orchestra.run.smithi142.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB]
2024-05-08T01:11:14.021 INFO:teuthology.orchestra.run.smithi142.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB]
2024-05-08T01:11:14.160 INFO:teuthology.orchestra.run.smithi142.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB]
2024-05-08T01:11:14.243 INFO:teuthology.orchestra.run.smithi142.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB]
2024-05-08T01:11:14.349 INFO:teuthology.orchestra.run.smithi142.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB]
2024-05-08T01:11:15.658 INFO:teuthology.orchestra.run.smithi142.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB]
2024-05-08T01:11:16.201 INFO:teuthology.orchestra.run.smithi142.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB]
2024-05-08T01:11:16.297 INFO:teuthology.orchestra.run.smithi142.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB]
2024-05-08T01:11:16.335 INFO:teuthology.orchestra.run.smithi142.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB]
2024-05-08T01:11:16.744 INFO:teuthology.orchestra.run.smithi142.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB]
2024-05-08T01:11:17.395 INFO:teuthology.orchestra.run.smithi142.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB]
2024-05-08T01:11:17.609 INFO:teuthology.orchestra.run.smithi142.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB]
2024-05-08T01:11:17.690 INFO:teuthology.orchestra.run.smithi142.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB]
2024-05-08T01:11:19.331 INFO:teuthology.orchestra.run.smithi142.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB]
2024-05-08T01:11:19.449 INFO:teuthology.orchestra.run.smithi142.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB]
2024-05-08T01:11:20.445 INFO:teuthology.orchestra.run.smithi142.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B]
2024-05-08T01:11:20.645 INFO:teuthology.orchestra.run.smithi142.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB]
2024-05-08T01:11:20.919 INFO:teuthology.orchestra.run.smithi142.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB]
2024-05-08T01:11:20.937 INFO:teuthology.orchestra.run.smithi142.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB]
2024-05-08T01:11:21.092 INFO:teuthology.orchestra.run.smithi142.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB]
2024-05-08T01:11:21.121 INFO:teuthology.orchestra.run.smithi142.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB]
2024-05-08T01:11:21.279 INFO:teuthology.orchestra.run.smithi142.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB]
2024-05-08T01:11:21.282 INFO:teuthology.orchestra.run.smithi142.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB]
2024-05-08T01:11:21.283 INFO:teuthology.orchestra.run.smithi142.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB]
2024-05-08T01:11:21.306 INFO:teuthology.orchestra.run.smithi142.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB]
2024-05-08T01:11:21.311 INFO:teuthology.orchestra.run.smithi142.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB]
2024-05-08T01:11:25.026 INFO:teuthology.orchestra.run.smithi142.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2529-ge0c1691a-1jammy [121 kB]
2024-05-08T01:11:25.028 INFO:teuthology.orchestra.run.smithi142.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2529-ge0c1691a-1jammy [38.9 kB]
2024-05-08T01:11:25.029 INFO:teuthology.orchestra.run.smithi142.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2529-ge0c1691a-1jammy [13.4 MB]
2024-05-08T01:11:27.470 INFO:teuthology.orchestra.run.smithi106.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2529-ge0c1691a-1jammy [14.0 kB]
2024-05-08T01:11:27.471 INFO:teuthology.orchestra.run.smithi106.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2529-ge0c1691a-1jammy [869 kB]
2024-05-08T01:11:27.802 INFO:teuthology.orchestra.run.smithi106.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2529-ge0c1691a-1jammy [2,437 kB]
2024-05-08T01:11:28.598 INFO:teuthology.orchestra.run.smithi106.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2529-ge0c1691a-1jammy [791 kB]
2024-05-08T01:11:28.867 INFO:teuthology.orchestra.run.smithi106.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2529-ge0c1691a-1jammy [148 kB]
2024-05-08T01:11:28.918 INFO:teuthology.orchestra.run.smithi106.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2529-ge0c1691a-1jammy [2,331 kB]
2024-05-08T01:11:29.735 INFO:teuthology.orchestra.run.smithi106.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2529-ge0c1691a-1jammy [8,625 kB]
2024-05-08T01:11:30.489 INFO:teuthology.orchestra.run.smithi142.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2529-ge0c1691a-1jammy [91.6 kB]
2024-05-08T01:11:30.491 INFO:teuthology.orchestra.run.smithi142.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.0.0-2529-ge0c1691a-1jammy [3,122 kB]
2024-05-08T01:11:32.140 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 183 MB in 29s (6,249 kB/s)
2024-05-08T01:11:32.271 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package liblttng-ust1:amd64.
2024-05-08T01:11:32.369 INFO:teuthology.orchestra.run.smithi142.stdout:(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 165990 files and directories currently installed.)
2024-05-08T01:11:32.374 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ...
2024-05-08T01:11:32.445 INFO:teuthology.orchestra.run.smithi106.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2529-ge0c1691a-1jammy [14.2 kB]
2024-05-08T01:11:32.445 INFO:teuthology.orchestra.run.smithi106.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2529-ge0c1691a-1jammy [50.7 MB]
2024-05-08T01:11:32.452 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ...
2024-05-08T01:11:32.723 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libdouble-conversion3:amd64.
2024-05-08T01:11:32.749 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ...
2024-05-08T01:11:32.789 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ...
2024-05-08T01:11:33.083 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libpcre2-16-0:amd64.
2024-05-08T01:11:33.109 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ...
2024-05-08T01:11:33.141 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ...
2024-05-08T01:11:33.595 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libqt5core5a:amd64.
2024-05-08T01:11:33.621 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ...
2024-05-08T01:11:33.669 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ...
2024-05-08T01:11:34.715 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libqt5dbus5:amd64.
2024-05-08T01:11:34.741 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ...
2024-05-08T01:11:34.781 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ...
2024-05-08T01:11:35.118 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libqt5network5:amd64.
2024-05-08T01:11:35.144 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ...
2024-05-08T01:11:35.185 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ...
2024-05-08T01:11:35.529 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libthrift-0.16.0:amd64.
2024-05-08T01:11:35.555 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ...
2024-05-08T01:11:35.596 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ...
2024-05-08T01:11:35.857 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package librados2.
2024-05-08T01:11:35.883 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../007-librados2_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:35.915 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking librados2 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:36.334 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package librbd1.
2024-05-08T01:11:36.361 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../008-librbd1_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:36.409 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking librbd1 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:36.854 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libcephfs2.
2024-05-08T01:11:36.880 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:36.920 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libcephfs2 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:37.366 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-rados.
2024-05-08T01:11:37.391 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:37.423 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-rados (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:37.742 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-ceph-argparse.
2024-05-08T01:11:37.769 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2529-ge0c1691a-1jammy_all.deb ...
2024-05-08T01:11:37.809 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-ceph-argparse (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:38.045 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-cephfs.
2024-05-08T01:11:38.060 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:38.103 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-cephfs (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:38.414 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-ceph-common.
2024-05-08T01:11:38.440 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2529-ge0c1691a-1jammy_all.deb ...
2024-05-08T01:11:38.481 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-ceph-common (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:38.809 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-wcwidth.
2024-05-08T01:11:38.835 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ...
2024-05-08T01:11:38.875 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ...
2024-05-08T01:11:39.161 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-prettytable.
2024-05-08T01:11:39.188 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ...
2024-05-08T01:11:39.219 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-prettytable (2.5.0-2) ...
2024-05-08T01:11:39.447 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-rbd.
2024-05-08T01:11:39.462 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:39.505 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-rbd (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:39.850 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package librdkafka1:amd64.
2024-05-08T01:11:39.876 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ...
2024-05-08T01:11:39.916 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ...
2024-05-08T01:11:40.186 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libreadline-dev:amd64.
2024-05-08T01:11:40.201 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ...
2024-05-08T01:11:40.244 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ...
2024-05-08T01:11:40.538 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package liblua5.3-dev:amd64.
2024-05-08T01:11:40.554 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ...
2024-05-08T01:11:40.588 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ...
2024-05-08T01:11:40.875 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package lua5.1.
2024-05-08T01:11:40.880 INFO:teuthology.orchestra.run.smithi106.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2529-ge0c1691a-1jammy [121 kB]
2024-05-08T01:11:40.882 INFO:teuthology.orchestra.run.smithi106.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2529-ge0c1691a-1jammy [38.9 kB]
2024-05-08T01:11:40.883 INFO:teuthology.orchestra.run.smithi106.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2529-ge0c1691a-1jammy [13.4 MB]
2024-05-08T01:11:40.901 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ...
2024-05-08T01:11:40.941 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ...
2024-05-08T01:11:41.253 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package lua-any.
2024-05-08T01:11:41.279 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ...
2024-05-08T01:11:41.319 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking lua-any (27ubuntu1) ...
2024-05-08T01:11:41.564 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package zip.
2024-05-08T01:11:41.590 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ...
2024-05-08T01:11:41.630 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking zip (3.0-12build2) ...
2024-05-08T01:11:41.933 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package unzip.
2024-05-08T01:11:41.959 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ...
2024-05-08T01:11:41.987 INFO:teuthology.orchestra.run.smithi106.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2529-ge0c1691a-1jammy [91.6 kB]
2024-05-08T01:11:41.989 INFO:teuthology.orchestra.run.smithi106.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.0.0-2529-ge0c1691a-1jammy [3,122 kB]
2024-05-08T01:11:41.991 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking unzip (6.0-26ubuntu3.2) ...
2024-05-08T01:11:42.504 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package luarocks.
2024-05-08T01:11:42.530 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ...
2024-05-08T01:11:42.570 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ...
2024-05-08T01:11:42.813 INFO:teuthology.orchestra.run.smithi106.stdout:Fetched 183 MB in 40s (4,599 kB/s)
2024-05-08T01:11:42.932 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package liblttng-ust1:amd64.
2024-05-08T01:11:42.953 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package librgw2.
2024-05-08T01:11:42.979 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../025-librgw2_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:42.989 INFO:teuthology.orchestra.run.smithi106.stdout:(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 165990 files and directories currently installed.)
2024-05-08T01:11:42.994 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ...
2024-05-08T01:11:43.019 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking librgw2 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:43.065 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ...
2024-05-08T01:11:43.411 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libdouble-conversion3:amd64.
2024-05-08T01:11:43.437 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ...
2024-05-08T01:11:43.477 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ...
2024-05-08T01:11:43.546 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-rgw.
2024-05-08T01:11:43.562 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:43.596 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-rgw (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:43.813 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libpcre2-16-0:amd64.
2024-05-08T01:11:43.840 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ...
2024-05-08T01:11:43.899 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package liboath0:amd64.
2024-05-08T01:11:43.926 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ...
2024-05-08T01:11:43.966 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ...
2024-05-08T01:11:44.044 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ...
2024-05-08T01:11:44.269 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libradosstriper1.
2024-05-08T01:11:44.296 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:44.335 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libradosstriper1 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:44.450 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libqt5core5a:amd64.
2024-05-08T01:11:44.476 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ...
2024-05-08T01:11:44.525 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ...
2024-05-08T01:11:44.756 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-common.
2024-05-08T01:11:44.782 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:44.822 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-common (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:44.912 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libqt5dbus5:amd64.
2024-05-08T01:11:44.938 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ...
2024-05-08T01:11:44.978 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ...
2024-05-08T01:11:45.323 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libqt5network5:amd64.
2024-05-08T01:11:45.349 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ...
2024-05-08T01:11:45.390 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ...
2024-05-08T01:11:45.726 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libthrift-0.16.0:amd64.
2024-05-08T01:11:45.742 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ...
2024-05-08T01:11:45.776 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ...
2024-05-08T01:11:46.079 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package librados2.
2024-05-08T01:11:46.105 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../007-librados2_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:46.118 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-base.
2024-05-08T01:11:46.145 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:46.146 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking librados2 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:46.195 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-base (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:46.615 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package librbd1.
2024-05-08T01:11:46.641 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../008-librbd1_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:46.682 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking librbd1 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:46.714 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-jaraco.functools.
2024-05-08T01:11:46.729 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ...
2024-05-08T01:11:46.772 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ...
2024-05-08T01:11:47.058 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-cheroot.
2024-05-08T01:11:47.085 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ...
2024-05-08T01:11:47.125 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ...
2024-05-08T01:11:47.143 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libcephfs2.
2024-05-08T01:11:47.169 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:47.210 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libcephfs2 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:47.471 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-rados.
2024-05-08T01:11:47.496 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:47.529 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-rados (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:47.614 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-jaraco.classes.
2024-05-08T01:11:47.638 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ...
2024-05-08T01:11:47.678 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ...
2024-05-08T01:11:47.857 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-ceph-argparse.
2024-05-08T01:11:47.883 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2529-ge0c1691a-1jammy_all.deb ...
2024-05-08T01:11:47.923 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-ceph-argparse (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:47.972 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-jaraco.text.
2024-05-08T01:11:47.999 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ...
2024-05-08T01:11:48.039 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-jaraco.text (3.6.0-2) ...
2024-05-08T01:11:48.176 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-cephfs.
2024-05-08T01:11:48.191 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:48.234 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-cephfs (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:48.283 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-jaraco.collections.
2024-05-08T01:11:48.299 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ...
2024-05-08T01:11:48.341 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ...
2024-05-08T01:11:48.529 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-ceph-common.
2024-05-08T01:11:48.544 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2529-ge0c1691a-1jammy_all.deb ...
2024-05-08T01:11:48.578 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-ceph-common (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:48.653 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-tempora.
2024-05-08T01:11:48.680 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ...
2024-05-08T01:11:48.719 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-tempora (4.1.2-1) ...
2024-05-08T01:11:48.848 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-wcwidth.
2024-05-08T01:11:48.863 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ...
2024-05-08T01:11:48.898 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ...
2024-05-08T01:11:49.022 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-portend.
2024-05-08T01:11:49.049 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ...
2024-05-08T01:11:49.089 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-portend (3.0.0-1) ...
2024-05-08T01:11:49.226 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-prettytable.
2024-05-08T01:11:49.252 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ...
2024-05-08T01:11:49.293 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-prettytable (2.5.0-2) ...
2024-05-08T01:11:49.333 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-zc.lockfile.
2024-05-08T01:11:49.360 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ...
2024-05-08T01:11:49.392 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-zc.lockfile (2.0-1) ...
2024-05-08T01:11:49.529 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-rbd.
2024-05-08T01:11:49.544 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:49.587 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-rbd (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:49.678 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-cherrypy3.
2024-05-08T01:11:49.704 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ...
2024-05-08T01:11:49.744 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ...
2024-05-08T01:11:49.906 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package librdkafka1:amd64.
2024-05-08T01:11:49.933 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ...
2024-05-08T01:11:49.965 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ...
2024-05-08T01:11:50.098 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-natsort.
2024-05-08T01:11:50.111 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ...
2024-05-08T01:11:50.147 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-natsort (8.0.2-1) ...
2024-05-08T01:11:50.218 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libreadline-dev:amd64.
2024-05-08T01:11:50.244 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ...
2024-05-08T01:11:50.284 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ...
2024-05-08T01:11:50.426 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-logutils.
2024-05-08T01:11:50.452 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ...
2024-05-08T01:11:50.484 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-logutils (0.3.3-8) ...
2024-05-08T01:11:50.604 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package liblua5.3-dev:amd64.
2024-05-08T01:11:50.618 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ...
2024-05-08T01:11:50.662 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ...
2024-05-08T01:11:50.795 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-markupsafe.
2024-05-08T01:11:50.822 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ...
2024-05-08T01:11:50.862 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ...
2024-05-08T01:11:50.990 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package lua5.1.
2024-05-08T01:11:51.017 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ...
2024-05-08T01:11:51.057 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ...
2024-05-08T01:11:51.148 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-mako.
2024-05-08T01:11:51.175 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ...
2024-05-08T01:11:51.214 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ...
2024-05-08T01:11:51.327 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package lua-any.
2024-05-08T01:11:51.342 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ...
2024-05-08T01:11:51.385 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking lua-any (27ubuntu1) ...
2024-05-08T01:11:51.534 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-simplegeneric.
2024-05-08T01:11:51.561 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ...
2024-05-08T01:11:51.601 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-simplegeneric (0.8.1-3) ...
2024-05-08T01:11:51.629 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package zip.
2024-05-08T01:11:51.645 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ...
2024-05-08T01:11:51.688 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking zip (3.0-12build2) ...
2024-05-08T01:11:51.879 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-singledispatch.
2024-05-08T01:11:51.906 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ...
2024-05-08T01:11:51.937 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ...
2024-05-08T01:11:52.008 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package unzip.
2024-05-08T01:11:52.034 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ...
2024-05-08T01:11:52.074 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking unzip (6.0-26ubuntu3.2) ...
2024-05-08T01:11:52.206 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-webob.
2024-05-08T01:11:52.233 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ...
2024-05-08T01:11:52.273 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-webob (1:1.8.6-1.1) ...
2024-05-08T01:11:52.437 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package luarocks.
2024-05-08T01:11:52.463 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ...
2024-05-08T01:11:52.495 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ...
2024-05-08T01:11:52.617 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-waitress.
2024-05-08T01:11:52.644 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ...
2024-05-08T01:11:52.696 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ...
2024-05-08T01:11:52.869 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package librgw2.
2024-05-08T01:11:52.896 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../025-librgw2_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:52.936 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking librgw2 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:53.012 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-tempita.
2024-05-08T01:11:53.039 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ...
2024-05-08T01:11:53.078 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ...
2024-05-08T01:11:53.356 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-paste.
2024-05-08T01:11:53.383 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ...
2024-05-08T01:11:53.423 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ...
2024-05-08T01:11:53.446 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-rgw.
2024-05-08T01:11:53.461 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:53.496 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-rgw (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:53.752 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python-pastedeploy-tpl.
2024-05-08T01:11:53.774 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package liboath0:amd64.
2024-05-08T01:11:53.778 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ...
2024-05-08T01:11:53.801 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ...
2024-05-08T01:11:53.817 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ...
2024-05-08T01:11:53.841 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ...
2024-05-08T01:11:54.104 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pastedeploy.
2024-05-08T01:11:54.131 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ...
2024-05-08T01:11:54.170 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pastedeploy (2.1.1-1) ...
2024-05-08T01:11:54.194 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libradosstriper1.
2024-05-08T01:11:54.221 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:54.261 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libradosstriper1 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:54.423 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-webtest.
2024-05-08T01:11:54.450 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ...
2024-05-08T01:11:54.481 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-webtest (2.0.35-1) ...
2024-05-08T01:11:54.707 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-common.
2024-05-08T01:11:54.732 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:54.774 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-common (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:54.785 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pecan.
2024-05-08T01:11:54.812 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ...
2024-05-08T01:11:54.851 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ...
2024-05-08T01:11:55.272 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-certifi.
2024-05-08T01:11:55.299 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ...
2024-05-08T01:11:55.389 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-certifi (2020.6.20-1) ...
2024-05-08T01:11:55.692 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-idna.
2024-05-08T01:11:55.718 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ...
2024-05-08T01:11:55.758 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-idna (3.3-1) ...
2024-05-08T01:11:56.020 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-urllib3.
2024-05-08T01:11:56.035 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ...
2024-05-08T01:11:56.078 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ...
2024-05-08T01:11:56.186 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-base.
2024-05-08T01:11:56.212 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:56.261 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-base (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:56.322 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-requests.
2024-05-08T01:11:56.336 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ...
2024-05-08T01:11:56.372 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ...
2024-05-08T01:11:56.675 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-werkzeug.
2024-05-08T01:11:56.702 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ...
2024-05-08T01:11:56.742 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ...
2024-05-08T01:11:56.840 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-jaraco.functools.
2024-05-08T01:11:56.867 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ...
2024-05-08T01:11:56.906 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ...
2024-05-08T01:11:56.995 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-modules-core.
2024-05-08T01:11:57.010 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2529-ge0c1691a-1jammy_all.deb ...
2024-05-08T01:11:57.053 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:57.209 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-cheroot.
2024-05-08T01:11:57.236 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ...
2024-05-08T01:11:57.276 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ...
2024-05-08T01:11:57.382 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libsqlite3-mod-ceph.
2024-05-08T01:11:57.398 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:57.432 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:57.581 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-jaraco.classes.
2024-05-08T01:11:57.608 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ...
2024-05-08T01:11:57.639 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ...
2024-05-08T01:11:57.744 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-bcrypt.
2024-05-08T01:11:57.770 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ...
2024-05-08T01:11:57.926 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ...
2024-05-08T01:11:57.959 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-jaraco.text.
2024-05-08T01:11:57.986 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ...
2024-05-08T01:11:58.025 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-jaraco.text (3.6.0-2) ...
2024-05-08T01:11:58.230 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-openssl.
2024-05-08T01:11:58.257 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ...
2024-05-08T01:11:58.296 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-openssl (21.0.0-1) ...
2024-05-08T01:11:58.320 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-jaraco.collections.
2024-05-08T01:11:58.346 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ...
2024-05-08T01:11:58.386 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ...
2024-05-08T01:11:58.599 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr.
2024-05-08T01:11:58.615 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:58.649 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:58.673 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-tempora.
2024-05-08T01:11:58.700 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ...
2024-05-08T01:11:58.739 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-tempora (4.1.2-1) ...
2024-05-08T01:11:58.984 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-portend.
2024-05-08T01:11:59.007 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ...
2024-05-08T01:11:59.044 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mon.
2024-05-08T01:11:59.050 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-portend (3.0.0-1) ...
2024-05-08T01:11:59.071 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:59.110 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mon (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:59.353 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-zc.lockfile.
2024-05-08T01:11:59.380 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ...
2024-05-08T01:11:59.420 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-zc.lockfile (2.0-1) ...
2024-05-08T01:11:59.691 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-osd.
2024-05-08T01:11:59.718 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:11:59.723 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-cherrypy3.
2024-05-08T01:11:59.749 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-osd (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:11:59.749 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ...
2024-05-08T01:11:59.789 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ...
2024-05-08T01:12:00.110 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-natsort.
2024-05-08T01:12:00.125 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ...
2024-05-08T01:12:00.159 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-natsort (8.0.2-1) ...
2024-05-08T01:12:00.471 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-logutils.
2024-05-08T01:12:00.498 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ...
2024-05-08T01:12:00.538 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-logutils (0.3.3-8) ...
2024-05-08T01:12:00.769 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph.
2024-05-08T01:12:00.784 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../066-ceph_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:00.826 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:00.841 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-markupsafe.
2024-05-08T01:12:00.868 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ...
2024-05-08T01:12:00.907 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ...
2024-05-08T01:12:01.129 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-fuse.
2024-05-08T01:12:01.156 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:01.196 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-fuse (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:01.210 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-mako.
2024-05-08T01:12:01.236 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ...
2024-05-08T01:12:01.301 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ...
2024-05-08T01:12:01.541 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mds.
2024-05-08T01:12:01.556 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:01.588 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-simplegeneric.
2024-05-08T01:12:01.590 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mds (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:01.604 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ...
2024-05-08T01:12:01.638 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-simplegeneric (0.8.1-3) ...
2024-05-08T01:12:01.908 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-singledispatch.
2024-05-08T01:12:01.934 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ...
2024-05-08T01:12:01.966 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ...
2024-05-08T01:12:01.968 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package cephadm.
2024-05-08T01:12:01.995 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../069-cephadm_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:02.035 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking cephadm (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:02.244 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-webob.
2024-05-08T01:12:02.270 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ...
2024-05-08T01:12:02.310 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-webob (1:1.8.6-1.1) ...
2024-05-08T01:12:02.346 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-asyncssh.
2024-05-08T01:12:02.373 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ...
2024-05-08T01:12:02.412 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-asyncssh (2.5.0-1) ...
2024-05-08T01:12:02.689 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-waitress.
2024-05-08T01:12:02.698 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python-babel-localedata.
2024-05-08T01:12:02.714 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ...
2024-05-08T01:12:02.715 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ...
2024-05-08T01:12:02.757 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ...
2024-05-08T01:12:02.758 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ...
2024-05-08T01:12:03.050 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-tempita.
2024-05-08T01:12:03.076 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ...
2024-05-08T01:12:03.116 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ...
2024-05-08T01:12:03.386 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-paste.
2024-05-08T01:12:03.412 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ...
2024-05-08T01:12:03.452 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ...
2024-05-08T01:12:03.737 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-babel.
2024-05-08T01:12:03.764 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ...
2024-05-08T01:12:03.789 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python-pastedeploy-tpl.
2024-05-08T01:12:03.803 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ...
2024-05-08T01:12:03.805 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ...
2024-05-08T01:12:03.847 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ...
2024-05-08T01:12:04.106 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-jinja2.
2024-05-08T01:12:04.108 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-pastedeploy.
2024-05-08T01:12:04.132 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ...
2024-05-08T01:12:04.133 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ...
2024-05-08T01:12:04.166 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-pastedeploy (2.1.1-1) ...
2024-05-08T01:12:04.172 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ...
2024-05-08T01:12:04.500 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-cephadm.
2024-05-08T01:12:04.527 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2529-ge0c1691a-1jammy_all.deb ...
2024-05-08T01:12:04.567 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:04.661 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-webtest.
2024-05-08T01:12:04.688 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ...
2024-05-08T01:12:04.728 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-webtest (2.0.35-1) ...
2024-05-08T01:12:04.912 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-repoze.lru.
2024-05-08T01:12:04.938 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ...
2024-05-08T01:12:04.978 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-repoze.lru (0.7-2) ...
2024-05-08T01:12:05.073 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-pecan.
2024-05-08T01:12:05.099 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ...
2024-05-08T01:12:05.139 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ...
2024-05-08T01:12:05.272 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-routes.
2024-05-08T01:12:05.288 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ...
2024-05-08T01:12:05.322 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ...
2024-05-08T01:12:05.494 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-certifi.
2024-05-08T01:12:05.520 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ...
2024-05-08T01:12:05.560 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-certifi (2020.6.20-1) ...
2024-05-08T01:12:05.625 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-dashboard.
2024-05-08T01:12:05.652 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2529-ge0c1691a-1jammy_all.deb ...
2024-05-08T01:12:05.692 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:05.889 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-idna.
2024-05-08T01:12:05.915 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ...
2024-05-08T01:12:05.963 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-idna (3.3-1) ...
2024-05-08T01:12:06.325 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-urllib3.
2024-05-08T01:12:06.351 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ...
2024-05-08T01:12:06.399 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ...
2024-05-08T01:12:06.786 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-requests.
2024-05-08T01:12:06.813 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ...
2024-05-08T01:12:06.861 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ...
2024-05-08T01:12:06.880 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-sklearn-lib:amd64.
2024-05-08T01:12:06.896 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ...
2024-05-08T01:12:06.938 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ...
2024-05-08T01:12:07.197 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-werkzeug.
2024-05-08T01:12:07.224 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ...
2024-05-08T01:12:07.264 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ...
2024-05-08T01:12:07.442 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-joblib.
2024-05-08T01:12:07.469 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ...
2024-05-08T01:12:07.508 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ...
2024-05-08T01:12:07.600 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-mgr-modules-core.
2024-05-08T01:12:07.615 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2529-ge0c1691a-1jammy_all.deb ...
2024-05-08T01:12:07.667 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:07.870 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-threadpoolctl.
2024-05-08T01:12:07.897 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ...
2024-05-08T01:12:07.937 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ...
2024-05-08T01:12:08.047 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libsqlite3-mod-ceph.
2024-05-08T01:12:08.074 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:08.113 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:08.281 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-sklearn.
2024-05-08T01:12:08.309 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ...
2024-05-08T01:12:08.347 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ...
2024-05-08T01:12:08.450 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-bcrypt.
2024-05-08T01:12:08.477 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ...
2024-05-08T01:12:08.516 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ...
2024-05-08T01:12:08.819 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-openssl.
2024-05-08T01:12:08.846 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ...
2024-05-08T01:12:08.886 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-openssl (21.0.0-1) ...
2024-05-08T01:12:08.998 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local.
2024-05-08T01:12:09.025 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2529-ge0c1691a-1jammy_all.deb ...
2024-05-08T01:12:09.149 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:09.189 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-mgr.
2024-05-08T01:12:09.216 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:09.247 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-mgr (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:09.784 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-mon.
2024-05-08T01:12:09.810 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:09.850 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-mon (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:10.061 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-cachetools.
2024-05-08T01:12:10.089 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ...
2024-05-08T01:12:10.127 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-cachetools (5.0.0-1) ...
2024-05-08T01:12:10.389 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pyasn1.
2024-05-08T01:12:10.417 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ...
2024-05-08T01:12:10.447 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pyasn1 (0.4.8-1) ...
2024-05-08T01:12:10.462 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-osd.
2024-05-08T01:12:10.488 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:10.528 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-osd (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:10.759 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pyasn1-modules.
2024-05-08T01:12:10.786 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ...
2024-05-08T01:12:10.825 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ...
2024-05-08T01:12:11.111 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-rsa.
2024-05-08T01:12:11.127 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ...
2024-05-08T01:12:11.170 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-rsa (4.8-1) ...
2024-05-08T01:12:11.498 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-google-auth.
2024-05-08T01:12:11.526 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ...
2024-05-08T01:12:11.565 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-google-auth (1.5.1-3) ...
2024-05-08T01:12:11.582 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph.
2024-05-08T01:12:11.597 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../066-ceph_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:11.640 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:11.868 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-requests-oauthlib.
2024-05-08T01:12:11.895 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ...
2024-05-08T01:12:11.934 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ...
2024-05-08T01:12:11.976 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-fuse.
2024-05-08T01:12:12.003 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:12.059 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-fuse (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:12.220 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-websocket.
2024-05-08T01:12:12.236 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ...
2024-05-08T01:12:12.270 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-websocket (1.2.3-1) ...
2024-05-08T01:12:12.396 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-mds.
2024-05-08T01:12:12.423 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:12.454 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-mds (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:12.590 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-kubernetes.
2024-05-08T01:12:12.606 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ...
2024-05-08T01:12:12.648 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ...
2024-05-08T01:12:12.849 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package cephadm.
2024-05-08T01:12:12.876 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../069-cephadm_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:12.915 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking cephadm (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:13.236 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-asyncssh.
2024-05-08T01:12:13.262 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ...
2024-05-08T01:12:13.301 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-asyncssh (2.5.0-1) ...
2024-05-08T01:12:13.479 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-k8sevents.
2024-05-08T01:12:13.495 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2529-ge0c1691a-1jammy_all.deb ...
2024-05-08T01:12:13.537 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:13.605 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python-babel-localedata.
2024-05-08T01:12:13.620 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ...
2024-05-08T01:12:13.654 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ...
2024-05-08T01:12:13.815 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libonig5:amd64.
2024-05-08T01:12:13.843 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ...
2024-05-08T01:12:13.873 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ...
2024-05-08T01:12:14.201 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libjq1:amd64.
2024-05-08T01:12:14.229 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ...
2024-05-08T01:12:14.267 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ...
2024-05-08T01:12:14.537 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package jq.
2024-05-08T01:12:14.553 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ...
2024-05-08T01:12:14.595 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking jq (1.6-2.1ubuntu3) ...
2024-05-08T01:12:14.718 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-babel.
2024-05-08T01:12:14.734 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ...
2024-05-08T01:12:14.767 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ...
2024-05-08T01:12:14.848 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package socat.
2024-05-08T01:12:14.876 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ...
2024-05-08T01:12:14.914 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ...
2024-05-08T01:12:15.071 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-jinja2.
2024-05-08T01:12:15.098 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ...
2024-05-08T01:12:15.128 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ...
2024-05-08T01:12:15.226 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package xmlstarlet.
2024-05-08T01:12:15.242 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ...
2024-05-08T01:12:15.284 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking xmlstarlet (1.6.1-2.1) ...
2024-05-08T01:12:15.440 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-mgr-cephadm.
2024-05-08T01:12:15.467 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2529-ge0c1691a-1jammy_all.deb ...
2024-05-08T01:12:15.498 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:15.655 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-test.
2024-05-08T01:12:15.683 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:15.721 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-test (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:15.835 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-repoze.lru.
2024-05-08T01:12:15.851 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ...
2024-05-08T01:12:15.885 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-repoze.lru (0.7-2) ...
2024-05-08T01:12:16.154 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-routes.
2024-05-08T01:12:16.170 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ...
2024-05-08T01:12:16.213 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ...
2024-05-08T01:12:16.566 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-mgr-dashboard.
2024-05-08T01:12:16.593 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2529-ge0c1691a-1jammy_all.deb ...
2024-05-08T01:12:16.632 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:17.817 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-volume.
2024-05-08T01:12:17.845 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2529-ge0c1691a-1jammy_all.deb ...
2024-05-08T01:12:17.875 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-volume (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:17.938 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-sklearn-lib:amd64.
2024-05-08T01:12:17.953 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ...
2024-05-08T01:12:17.997 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ...
2024-05-08T01:12:18.146 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libcephfs-dev.
2024-05-08T01:12:18.161 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:18.195 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libcephfs-dev (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:18.490 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package lua-socket:amd64.
2024-05-08T01:12:18.499 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-joblib.
2024-05-08T01:12:18.517 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ...
2024-05-08T01:12:18.526 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ...
2024-05-08T01:12:18.556 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ...
2024-05-08T01:12:18.566 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ...
2024-05-08T01:12:18.836 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package lua-sec:amd64.
2024-05-08T01:12:18.863 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ...
2024-05-08T01:12:18.894 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ...
2024-05-08T01:12:18.945 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-threadpoolctl.
2024-05-08T01:12:18.971 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ...
2024-05-08T01:12:19.011 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ...
2024-05-08T01:12:19.248 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package nvme-cli.
2024-05-08T01:12:19.275 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ...
2024-05-08T01:12:19.314 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ...
2024-05-08T01:12:19.339 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-sklearn.
2024-05-08T01:12:19.366 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ...
2024-05-08T01:12:19.414 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ...
2024-05-08T01:12:19.711 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package pkg-config.
2024-05-08T01:12:19.738 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ...
2024-05-08T01:12:19.777 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ...
2024-05-08T01:12:20.030 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python-asyncssh-doc.
2024-05-08T01:12:20.046 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ...
2024-05-08T01:12:20.097 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ...
2024-05-08T01:12:20.114 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local.
2024-05-08T01:12:20.141 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2529-ge0c1691a-1jammy_all.deb ...
2024-05-08T01:12:20.283 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:20.508 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-iniconfig.
2024-05-08T01:12:20.536 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ...
2024-05-08T01:12:20.574 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-iniconfig (1.1.1-2) ...
2024-05-08T01:12:20.886 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pastescript.
2024-05-08T01:12:20.913 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ...
2024-05-08T01:12:20.952 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pastescript (2.0.2-4) ...
2024-05-08T01:12:21.253 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-cachetools.
2024-05-08T01:12:21.280 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ...
2024-05-08T01:12:21.281 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pluggy.
2024-05-08T01:12:21.308 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ...
2024-05-08T01:12:21.320 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-cachetools (5.0.0-1) ...
2024-05-08T01:12:21.339 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pluggy (0.13.0-7.1) ...
2024-05-08T01:12:21.600 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-psutil.
2024-05-08T01:12:21.614 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-pyasn1.
2024-05-08T01:12:21.627 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ...
2024-05-08T01:12:21.642 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ...
2024-05-08T01:12:21.666 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-psutil (5.9.0-1build1) ...
2024-05-08T01:12:21.681 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-pyasn1 (0.4.8-1) ...
2024-05-08T01:12:21.993 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-pyasn1-modules.
2024-05-08T01:12:22.003 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-py.
2024-05-08T01:12:22.019 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ...
2024-05-08T01:12:22.030 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ...
2024-05-08T01:12:22.059 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ...
2024-05-08T01:12:22.069 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-py (1.10.0-1) ...
2024-05-08T01:12:22.371 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-rsa.
2024-05-08T01:12:22.373 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pygments.
2024-05-08T01:12:22.398 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ...
2024-05-08T01:12:22.400 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ...
2024-05-08T01:12:22.431 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ...
2024-05-08T01:12:22.437 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-rsa (4.8-1) ...
2024-05-08T01:12:22.749 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-google-auth.
2024-05-08T01:12:22.777 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ...
2024-05-08T01:12:22.824 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-google-auth (1.5.1-3) ...
2024-05-08T01:12:22.893 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pyinotify.
2024-05-08T01:12:22.921 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ...
2024-05-08T01:12:22.959 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ...
2024-05-08T01:12:23.144 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-requests-oauthlib.
2024-05-08T01:12:23.171 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ...
2024-05-08T01:12:23.210 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ...
2024-05-08T01:12:23.270 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-toml.
2024-05-08T01:12:23.286 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ...
2024-05-08T01:12:23.329 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-toml (0.10.2-1) ...
2024-05-08T01:12:23.480 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-websocket.
2024-05-08T01:12:23.505 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ...
2024-05-08T01:12:23.538 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-websocket (1.2.3-1) ...
2024-05-08T01:12:23.607 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pytest.
2024-05-08T01:12:23.634 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ...
2024-05-08T01:12:23.673 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ...
2024-05-08T01:12:23.858 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-kubernetes.
2024-05-08T01:12:23.886 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ...
2024-05-08T01:12:23.925 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ...
2024-05-08T01:12:24.002 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-simplejson.
2024-05-08T01:12:24.029 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ...
2024-05-08T01:12:24.069 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-simplejson (3.17.6-1build1) ...
2024-05-08T01:12:24.354 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package qttranslations5-l10n.
2024-05-08T01:12:24.382 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ...
2024-05-08T01:12:24.420 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ...
2024-05-08T01:12:24.597 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-mgr-k8sevents.
2024-05-08T01:12:24.625 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2529-ge0c1691a-1jammy_all.deb ...
2024-05-08T01:12:24.655 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:25.108 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package radosgw.
2024-05-08T01:12:25.136 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../116-radosgw_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:25.158 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libonig5:amd64.
2024-05-08T01:12:25.174 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking radosgw (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:25.186 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ...
2024-05-08T01:12:25.225 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ...
2024-05-08T01:12:25.570 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libjq1:amd64.
2024-05-08T01:12:25.597 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ...
2024-05-08T01:12:25.636 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ...
2024-05-08T01:12:25.905 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package jq.
2024-05-08T01:12:25.933 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ...
2024-05-08T01:12:25.972 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking jq (1.6-2.1ubuntu3) ...
2024-05-08T01:12:26.018 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package rbd-fuse.
2024-05-08T01:12:26.046 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:26.085 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking rbd-fuse (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:26.216 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package socat.
2024-05-08T01:12:26.232 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ...
2024-05-08T01:12:26.266 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ...
2024-05-08T01:12:26.371 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package rbd-mirror.
2024-05-08T01:12:26.387 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../118-rbd-mirror_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:26.421 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking rbd-mirror (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:26.577 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package xmlstarlet.
2024-05-08T01:12:26.594 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ...
2024-05-08T01:12:26.636 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking xmlstarlet (1.6.1-2.1) ...
2024-05-08T01:12:26.867 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-iniconfig (1.1.1-2) ...
2024-05-08T01:12:26.982 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-test.
2024-05-08T01:12:27.010 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:27.048 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-test (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:27.110 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ...
2024-05-08T01:12:27.235 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ...
2024-05-08T01:12:27.497 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service.
2024-05-08T01:12:27.812 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service.
2024-05-08T01:12:28.231 INFO:teuthology.orchestra.run.smithi142.stdout:nvmf-connect.target is a disabled or a static unit, not starting it.
2024-05-08T01:12:28.295 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up cephadm (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:28.828 INFO:teuthology.orchestra.run.smithi142.stdout:Adding system user cephadm....done
2024-05-08T01:12:28.976 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ...
2024-05-08T01:12:29.244 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-jaraco.classes (3.2.1-3) ...
2024-05-08T01:12:29.328 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-volume.
2024-05-08T01:12:29.355 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2529-ge0c1691a-1jammy_all.deb ...
2024-05-08T01:12:29.394 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-volume (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:29.495 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python-asyncssh-doc (2.5.0-1) ...
2024-05-08T01:12:29.621 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-jaraco.functools (3.4.0-2) ...
2024-05-08T01:12:29.715 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libcephfs-dev.
2024-05-08T01:12:29.732 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:29.773 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libcephfs-dev (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:29.898 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-repoze.lru (0.7-2) ...
2024-05-08T01:12:30.143 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package lua-socket:amd64.
2024-05-08T01:12:30.148 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ...
2024-05-08T01:12:30.170 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ...
2024-05-08T01:12:30.209 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ...
2024-05-08T01:12:30.275 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-py (1.10.0-1) ...
2024-05-08T01:12:30.514 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package lua-sec:amd64.
2024-05-08T01:12:30.541 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ...
2024-05-08T01:12:30.572 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ...
2024-05-08T01:12:30.576 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ...
2024-05-08T01:12:30.934 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package nvme-cli.
2024-05-08T01:12:30.936 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-cachetools (5.0.0-1) ...
2024-05-08T01:12:30.962 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ...
2024-05-08T01:12:31.001 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ...
2024-05-08T01:12:31.196 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-openssl (21.0.0-1) ...
2024-05-08T01:12:31.426 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package pkg-config.
2024-05-08T01:12:31.450 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ...
2024-05-08T01:12:31.464 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up unzip (6.0-26ubuntu3.2) ...
2024-05-08T01:12:31.489 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ...
2024-05-08T01:12:31.724 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-bcrypt (3.2.0-1build1) ...
2024-05-08T01:12:31.742 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python-asyncssh-doc.
2024-05-08T01:12:31.758 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ...
2024-05-08T01:12:31.800 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ...
2024-05-08T01:12:31.966 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pyinotify (0.9.6-1.3) ...
2024-05-08T01:12:32.217 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-threadpoolctl (3.1.0-1) ...
2024-05-08T01:12:32.228 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-iniconfig.
2024-05-08T01:12:32.256 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ...
2024-05-08T01:12:32.295 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-iniconfig (1.1.1-2) ...
2024-05-08T01:12:32.460 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-ceph-argparse (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:32.556 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-pastescript.
2024-05-08T01:12:32.583 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ...
2024-05-08T01:12:32.614 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-pastescript (2.0.2-4) ...
2024-05-08T01:12:32.719 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ...
2024-05-08T01:12:32.845 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ...
2024-05-08T01:12:32.918 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-pluggy.
2024-05-08T01:12:32.946 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ...
2024-05-08T01:12:32.975 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ...
2024-05-08T01:12:32.984 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-pluggy (0.13.0-7.1) ...
2024-05-08T01:12:33.089 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-markupsafe (2.0.1-2build1) ...
2024-05-08T01:12:33.295 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-psutil.
2024-05-08T01:12:33.323 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ...
2024-05-08T01:12:33.340 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up lua5.1 (5.1.5-8.1build4) ...
2024-05-08T01:12:33.362 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-psutil (5.9.0-1build1) ...
2024-05-08T01:12:33.682 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-py.
2024-05-08T01:12:33.689 INFO:teuthology.orchestra.run.smithi142.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode
2024-05-08T01:12:33.709 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ...
2024-05-08T01:12:33.721 INFO:teuthology.orchestra.run.smithi142.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode
2024-05-08T01:12:33.740 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-py (1.10.0-1) ...
2024-05-08T01:12:33.785 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ...
2024-05-08T01:12:33.911 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-psutil (5.9.0-1build1) ...
2024-05-08T01:12:34.002 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-pygments.
2024-05-08T01:12:34.029 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ...
2024-05-08T01:12:34.060 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ...
2024-05-08T01:12:34.270 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ...
2024-05-08T01:12:34.396 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-natsort (8.0.2-1) ...
2024-05-08T01:12:34.514 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-pyinotify.
2024-05-08T01:12:34.542 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ...
2024-05-08T01:12:34.580 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ...
2024-05-08T01:12:34.664 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ...
2024-05-08T01:12:34.877 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-toml.
2024-05-08T01:12:34.904 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ...
2024-05-08T01:12:34.907 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-simplejson (3.17.6-1build1) ...
2024-05-08T01:12:34.943 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-toml (0.10.2-1) ...
2024-05-08T01:12:35.213 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-pytest.
2024-05-08T01:12:35.216 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up zip (3.0-12build2) ...
2024-05-08T01:12:35.229 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ...
2024-05-08T01:12:35.271 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ...
2024-05-08T01:12:35.342 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ...
2024-05-08T01:12:35.616 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-simplejson.
2024-05-08T01:12:35.644 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ...
2024-05-08T01:12:35.682 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-simplejson (3.17.6-1build1) ...
2024-05-08T01:12:35.961 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package qttranslations5-l10n.
2024-05-08T01:12:35.989 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ...
2024-05-08T01:12:36.027 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ...
2024-05-08T01:12:36.027 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ...
2024-05-08T01:12:36.279 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ...
2024-05-08T01:12:36.405 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up qttranslations5-l10n (5.15.3-1) ...
2024-05-08T01:12:36.539 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ...
2024-05-08T01:12:36.756 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package radosgw.
2024-05-08T01:12:36.784 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../116-radosgw_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:36.814 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking radosgw (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:36.832 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-asyncssh (2.5.0-1) ...
2024-05-08T01:12:37.208 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-certifi (2020.6.20-1) ...
2024-05-08T01:12:37.451 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ...
2024-05-08T01:12:37.600 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package rbd-fuse.
2024-05-08T01:12:37.628 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:37.666 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking rbd-fuse (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:37.844 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ...
2024-05-08T01:12:38.003 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package rbd-mirror.
2024-05-08T01:12:38.031 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../118-rbd-mirror_19.0.0-2529-ge0c1691a-1jammy_amd64.deb ...
2024-05-08T01:12:38.069 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking rbd-mirror (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:38.137 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ...
2024-05-08T01:12:38.505 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-jaraco.text (3.6.0-2) ...
2024-05-08T01:12:38.532 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-iniconfig (1.1.1-2) ...
2024-05-08T01:12:38.748 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up socat (1.7.4.1-3ubuntu4) ...
2024-05-08T01:12:38.799 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ...
2024-05-08T01:12:38.857 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-ceph-common (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:38.900 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ...
2024-05-08T01:12:39.167 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-idna (3.3-1) ...
2024-05-08T01:12:39.169 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service.
2024-05-08T01:12:39.459 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ...
2024-05-08T01:12:39.480 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service.
2024-05-08T01:12:39.918 INFO:teuthology.orchestra.run.smithi106.stdout:nvmf-connect.target is a disabled or a static unit, not starting it.
2024-05-08T01:12:39.986 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up cephadm (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:40.607 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ...
2024-05-08T01:12:40.802 INFO:teuthology.orchestra.run.smithi106.stdout:Adding system user cephadm....done
2024-05-08T01:12:40.833 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ...
2024-05-08T01:12:40.943 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ...
2024-05-08T01:12:40.943 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-toml (0.10.2-1) ...
2024-05-08T01:12:41.202 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ...
2024-05-08T01:12:41.242 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-jaraco.classes (3.2.1-3) ...
2024-05-08T01:12:41.328 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up xmlstarlet (1.6.1-2.1) ...
2024-05-08T01:12:41.454 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ...
2024-05-08T01:12:41.486 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python-asyncssh-doc (2.5.0-1) ...
2024-05-08T01:12:41.612 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-jaraco.functools (3.4.0-2) ...
2024-05-08T01:12:41.772 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pluggy (0.13.0-7.1) ...
2024-05-08T01:12:41.871 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-repoze.lru (0.7-2) ...
2024-05-08T01:12:42.023 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-zc.lockfile (2.0-1) ...
2024-05-08T01:12:42.131 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ...
2024-05-08T01:12:42.257 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-py (1.10.0-1) ...
2024-05-08T01:12:42.274 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ...
2024-05-08T01:12:42.384 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pyasn1 (0.4.8-1) ...
2024-05-08T01:12:42.550 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ...
2024-05-08T01:12:42.669 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-singledispatch (3.4.0.3-3) ...
2024-05-08T01:12:42.902 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-cachetools (5.0.0-1) ...
2024-05-08T01:12:42.919 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-logutils (0.3.3-8) ...
2024-05-08T01:12:43.154 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-openssl (21.0.0-1) ...
2024-05-08T01:12:43.204 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-tempora (4.1.2-1) ...
2024-05-08T01:12:43.421 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up unzip (6.0-26ubuntu3.2) ...
2024-05-08T01:12:43.465 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-simplegeneric (0.8.1-3) ...
2024-05-08T01:12:43.681 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-bcrypt (3.2.0-1build1) ...
2024-05-08T01:12:43.708 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-prettytable (2.5.0-2) ...
2024-05-08T01:12:43.932 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-pyinotify (0.9.6-1.3) ...
2024-05-08T01:12:43.976 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ...
2024-05-08T01:12:44.175 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-threadpoolctl (3.1.0-1) ...
2024-05-08T01:12:44.184 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-websocket (1.2.3-1) ...
2024-05-08T01:12:44.418 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-ceph-argparse (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:44.685 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ...
2024-05-08T01:12:44.728 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ...
2024-05-08T01:12:44.811 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ...
2024-05-08T01:12:44.854 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ...
2024-05-08T01:12:44.938 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ...
2024-05-08T01:12:45.063 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-markupsafe (2.0.1-2build1) ...
2024-05-08T01:12:45.146 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-webob (1:1.8.6-1.1) ...
2024-05-08T01:12:45.322 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up lua5.1 (5.1.5-8.1build4) ...
2024-05-08T01:12:45.448 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-jaraco.collections (3.4.0-2) ...
2024-05-08T01:12:45.701 INFO:teuthology.orchestra.run.smithi106.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode
2024-05-08T01:12:45.707 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ...
2024-05-08T01:12:45.728 INFO:teuthology.orchestra.run.smithi106.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode
2024-05-08T01:12:45.793 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ...
2024-05-08T01:12:45.808 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up lua-sec:amd64 (1.0.2-1) ...
2024-05-08T01:12:45.909 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ...
2024-05-08T01:12:45.919 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-psutil (5.9.0-1build1) ...
2024-05-08T01:12:45.998 INFO:teuthology.orchestra.run.smithi142.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode
2024-05-08T01:12:46.236 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ...
2024-05-08T01:12:46.278 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ...
2024-05-08T01:12:46.362 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ...
2024-05-08T01:12:46.404 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-natsort (8.0.2-1) ...
2024-05-08T01:12:46.672 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ...
2024-05-08T01:12:46.759 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pastedeploy (2.1.1-1) ...
2024-05-08T01:12:46.940 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-simplejson (3.17.6-1build1) ...
2024-05-08T01:12:47.023 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up lua-any (27ubuntu1) ...
2024-05-08T01:12:47.149 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-portend (3.0.0-1) ...
2024-05-08T01:12:47.233 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up zip (3.0-12build2) ...
2024-05-08T01:12:47.351 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ...
2024-05-08T01:12:47.425 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ...
2024-05-08T01:12:47.551 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ...
2024-05-08T01:12:47.877 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ...
2024-05-08T01:12:48.002 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ...
2024-05-08T01:12:48.195 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ...
2024-05-08T01:12:48.254 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ...
2024-05-08T01:12:48.379 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up qttranslations5-l10n (5.15.3-1) ...
2024-05-08T01:12:48.471 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up jq (1.6-2.1ubuntu3) ...
2024-05-08T01:12:48.505 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ...
2024-05-08T01:12:48.580 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-rsa (4.8-1) ...
2024-05-08T01:12:48.807 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-asyncssh (2.5.0-1) ...
2024-05-08T01:12:48.840 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-webtest (2.0.35-1) ...
2024-05-08T01:12:49.108 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ...
2024-05-08T01:12:49.158 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-certifi (2020.6.20-1) ...
2024-05-08T01:12:49.392 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ...
2024-05-08T01:12:49.478 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-cherrypy3 (18.6.1-4) ...
2024-05-08T01:12:49.769 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ...
2024-05-08T01:12:49.863 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pastescript (2.0.2-4) ...
2024-05-08T01:12:50.062 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ...
2024-05-08T01:12:50.139 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ...
2024-05-08T01:12:50.413 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-jaraco.text (3.6.0-2) ...
2024-05-08T01:12:50.459 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ...
2024-05-08T01:12:50.584 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up librados2 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:50.672 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up socat (1.7.4.1-3ubuntu4) ...
2024-05-08T01:12:50.711 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:50.790 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-ceph-common (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:50.836 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up luarocks (3.8.0+dfsg1-1) ...
2024-05-08T01:12:51.075 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-idna (3.3-1) ...
2024-05-08T01:12:51.139 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libcephfs2 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:51.257 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libradosstriper1 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:51.376 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ...
2024-05-08T01:12:51.383 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-google-auth (1.5.1-3) ...
2024-05-08T01:12:51.660 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up librbd1 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:51.786 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-modules-core (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:51.912 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-fuse (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:52.152 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target.
2024-05-08T01:12:52.153 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target.
2024-05-08T01:12:52.590 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ...
2024-05-08T01:12:52.613 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libcephfs-dev (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:52.722 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-rados (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:52.823 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up librgw2 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:52.825 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ...
2024-05-08T01:12:52.933 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ...
2024-05-08T01:12:52.942 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-toml (0.10.2-1) ...
2024-05-08T01:12:53.202 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ...
2024-05-08T01:12:53.328 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up xmlstarlet (1.6.1-2.1) ...
2024-05-08T01:12:53.471 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ...
2024-05-08T01:12:53.755 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-pluggy (0.13.0-7.1) ...
2024-05-08T01:12:54.015 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-zc.lockfile (2.0-1) ...
2024-05-08T01:12:54.086 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-rbd (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:54.212 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up rbd-fuse (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:54.257 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ...
2024-05-08T01:12:54.321 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-rgw (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:54.375 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-pyasn1 (0.4.8-1) ...
2024-05-08T01:12:54.422 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-cephfs (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:54.532 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-common (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:54.702 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-singledispatch (3.4.0.3-3) ...
2024-05-08T01:12:54.961 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-logutils (0.3.3-8) ...
2024-05-08T01:12:55.058 INFO:teuthology.orchestra.run.smithi142.stdout:Adding group ceph....done
2024-05-08T01:12:55.246 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-tempora (4.1.2-1) ...
2024-05-08T01:12:55.443 INFO:teuthology.orchestra.run.smithi142.stdout:Adding system user ceph....done
2024-05-08T01:12:55.497 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-simplegeneric (0.8.1-3) ...
2024-05-08T01:12:55.603 INFO:teuthology.orchestra.run.smithi142.stdout:Setting system user ceph properties....done
2024-05-08T01:12:55.615 INFO:teuthology.orchestra.run.smithi142.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory
2024-05-08T01:12:55.748 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-prettytable (2.5.0-2) ...
2024-05-08T01:12:55.752 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target.
2024-05-08T01:12:56.101 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service.
2024-05-08T01:12:56.241 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ...
2024-05-08T01:12:56.350 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-websocket (1.2.3-1) ...
2024-05-08T01:12:56.616 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-test (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:56.634 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ...
2024-05-08T01:12:56.751 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up radosgw (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:56.761 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ...
2024-05-08T01:12:57.054 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-webob (1:1.8.6-1.1) ...
2024-05-08T01:12:57.346 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-jaraco.collections (3.4.0-2) ...
2024-05-08T01:12:57.539 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target.
2024-05-08T01:12:57.539 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target.
2024-05-08T01:12:57.614 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ...
2024-05-08T01:12:57.740 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up lua-sec:amd64 (1.0.2-1) ...
2024-05-08T01:12:57.858 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ...
2024-05-08T01:12:57.931 INFO:teuthology.orchestra.run.smithi106.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode
2024-05-08T01:12:57.995 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-base (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:58.168 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ...
2024-05-08T01:12:58.294 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ...
2024-05-08T01:12:58.324 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service.
2024-05-08T01:12:58.687 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-pastedeploy (2.1.1-1) ...
2024-05-08T01:12:58.771 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mds (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:58.946 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up lua-any (27ubuntu1) ...
2024-05-08T01:12:58.984 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target.
2024-05-08T01:12:58.984 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target.
2024-05-08T01:12:59.072 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-portend (3.0.0-1) ...
2024-05-08T01:12:59.323 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ...
2024-05-08T01:12:59.432 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up rbd-mirror (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:12:59.433 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ...
2024-05-08T01:12:59.641 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target.
2024-05-08T01:12:59.641 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target.
2024-05-08T01:12:59.734 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ...
2024-05-08T01:13:00.035 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ...
2024-05-08T01:13:00.287 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:00.311 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up jq (1.6-2.1ubuntu3) ...
2024-05-08T01:13:00.437 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-rsa (4.8-1) ...
2024-05-08T01:13:00.500 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target.
2024-05-08T01:13:00.500 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target.
2024-05-08T01:13:00.705 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-webtest (2.0.35-1) ...
2024-05-08T01:13:00.914 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-osd (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:00.981 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ...
2024-05-08T01:13:01.177 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target.
2024-05-08T01:13:01.178 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target.
2024-05-08T01:13:01.309 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-cherrypy3 (18.6.1-4) ...
2024-05-08T01:13:01.607 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:01.695 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-pastescript (2.0.2-4) ...
2024-05-08T01:13:01.733 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:01.893 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mon (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:01.979 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ...
2024-05-08T01:13:02.104 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target.
2024-05-08T01:13:02.104 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target.
2024-05-08T01:13:02.332 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ...
2024-05-08T01:13:02.441 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up librados2 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:02.542 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:02.604 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-cephadm (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:02.652 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up luarocks (3.8.0+dfsg1-1) ...
2024-05-08T01:13:02.853 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:02.979 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-dashboard (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:02.988 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libcephfs2 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:03.114 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libradosstriper1 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:03.130 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-volume (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:03.240 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-google-auth (1.5.1-3) ...
2024-05-08T01:13:03.483 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ...
2024-05-08T01:13:03.516 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up librbd1 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:03.592 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
2024-05-08T01:13:03.642 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-mgr-modules-core (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:03.745 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for man-db (2.10.2-1) ...
2024-05-08T01:13:03.768 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-fuse (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:03.981 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target.
2024-05-08T01:13:03.981 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target.
2024-05-08T01:13:04.453 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libcephfs-dev (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:04.579 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-rados (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:04.705 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up librgw2 (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:04.787 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for install-info (6.8-4build1) ...
2024-05-08T01:13:04.831 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ...
2024-05-08T01:13:05.434 INFO:teuthology.orchestra.run.smithi142.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead.
2024-05-08T01:13:05.440 DEBUG:teuthology.parallel:result is None
2024-05-08T01:13:06.043 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-rbd (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:06.169 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up rbd-fuse (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:06.295 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-rgw (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:06.438 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-cephfs (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:06.564 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-common (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:07.107 INFO:teuthology.orchestra.run.smithi106.stdout:Adding group ceph....done
2024-05-08T01:13:07.491 INFO:teuthology.orchestra.run.smithi106.stdout:Adding system user ceph....done
2024-05-08T01:13:07.660 INFO:teuthology.orchestra.run.smithi106.stdout:Setting system user ceph properties....done
2024-05-08T01:13:07.679 INFO:teuthology.orchestra.run.smithi106.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory
2024-05-08T01:13:07.819 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target.
2024-05-08T01:13:08.179 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service.
2024-05-08T01:13:08.673 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-test (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:08.791 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up radosgw (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:09.232 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target.
2024-05-08T01:13:09.232 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target.
2024-05-08T01:13:09.694 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-base (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:10.045 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service.
2024-05-08T01:13:10.553 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-mds (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:10.769 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target.
2024-05-08T01:13:10.770 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target.
2024-05-08T01:13:11.230 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up rbd-mirror (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:11.464 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target.
2024-05-08T01:13:11.465 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target.
2024-05-08T01:13:11.953 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-mgr (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:12.182 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target.
2024-05-08T01:13:12.182 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target.
2024-05-08T01:13:12.687 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-osd (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:12.974 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target.
2024-05-08T01:13:12.975 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target.
2024-05-08T01:13:13.497 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:13.623 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:13.799 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-mon (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:13.983 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target.
2024-05-08T01:13:13.984 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target.
2024-05-08T01:13:14.502 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-mgr-cephadm (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:14.722 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:14.920 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-mgr-dashboard (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:15.202 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-volume (19.0.0-2529-ge0c1691a-1jammy) ...
2024-05-08T01:13:15.632 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ...
2024-05-08T01:13:15.725 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
2024-05-08T01:13:15.877 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for man-db (2.10.2-1) ...
2024-05-08T01:13:16.635 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for install-info (6.8-4build1) ...
2024-05-08T01:13:17.262 INFO:teuthology.orchestra.run.smithi106.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead.
2024-05-08T01:13:17.265 DEBUG:teuthology.parallel:result is None
2024-05-08T01:13:17.266 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e0c1691aade07c0b78ccff9531f65d3c80e3e220
2024-05-08T01:13:17.499 DEBUG:teuthology.orchestra.run.smithi106:> dpkg-query -W -f '${Version}' ceph
2024-05-08T01:13:17.518 INFO:teuthology.orchestra.run.smithi106.stdout:19.0.0-2529-ge0c1691a-1jammy
2024-05-08T01:13:17.519 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2529-ge0c1691a-1jammy
2024-05-08T01:13:17.519 INFO:teuthology.task.install:The correct ceph version 19.0.0-2529-ge0c1691a-1jammy is installed.
2024-05-08T01:13:17.521 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e0c1691aade07c0b78ccff9531f65d3c80e3e220
2024-05-08T01:13:17.655 DEBUG:teuthology.orchestra.run.smithi142:> dpkg-query -W -f '${Version}' ceph
2024-05-08T01:13:17.674 INFO:teuthology.orchestra.run.smithi142.stdout:19.0.0-2529-ge0c1691a-1jammy
2024-05-08T01:13:17.674 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2529-ge0c1691a-1jammy
2024-05-08T01:13:17.674 INFO:teuthology.task.install:The correct ceph version 19.0.0-2529-ge0c1691a-1jammy is installed.
2024-05-08T01:13:17.676 INFO:teuthology.task.install.util:Shipping valgrind.supp...
2024-05-08T01:13:17.676 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:17.676 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp
2024-05-08T01:13:17.691 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:17.691 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp
2024-05-08T01:13:17.731 INFO:teuthology.task.install.util:Shipping 'daemon-helper'...
2024-05-08T01:13:17.731 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:17.731 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/daemon-helper
2024-05-08T01:13:17.745 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/daemon-helper
2024-05-08T01:13:17.801 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:17.801 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/daemon-helper
2024-05-08T01:13:17.815 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/daemon-helper
2024-05-08T01:13:17.870 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'...
2024-05-08T01:13:17.870 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:17.870 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/adjust-ulimits
2024-05-08T01:13:17.885 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/adjust-ulimits
2024-05-08T01:13:17.937 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:17.937 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/adjust-ulimits
2024-05-08T01:13:17.950 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/adjust-ulimits
2024-05-08T01:13:18.003 INFO:teuthology.task.install.util:Shipping 'stdin-killer'...
2024-05-08T01:13:18.004 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:18.004 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/stdin-killer
2024-05-08T01:13:18.017 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/stdin-killer
2024-05-08T01:13:18.073 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:18.074 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/stdin-killer
2024-05-08T01:13:18.087 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/stdin-killer
2024-05-08T01:13:18.142 INFO:teuthology.run_tasks:Running task ceph...
2024-05-08T01:13:18.224 INFO:tasks.ceph:Making ceph log dir writeable by non-root...
2024-05-08T01:13:18.225 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 777 /var/log/ceph
2024-05-08T01:13:18.227 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 777 /var/log/ceph
2024-05-08T01:13:18.240 INFO:tasks.ceph:Disabling ceph logrotate...
2024-05-08T01:13:18.241 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /etc/logrotate.d/ceph
2024-05-08T01:13:18.283 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/logrotate.d/ceph
2024-05-08T01:13:18.295 INFO:tasks.ceph:Creating extra log directories...
2024-05-08T01:13:18.295 DEBUG:teuthology.orchestra.run.smithi106:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger
2024-05-08T01:13:18.340 DEBUG:teuthology.orchestra.run.smithi142:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger
2024-05-08T01:13:18.365 INFO:tasks.ceph:Creating ceph cluster cluster1...
2024-05-08T01:13:18.366 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True}
2024-05-08T01:13:18.366 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696665', 'branch': 'wip-dis-testing-squid', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-policy-none}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7696665', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror.sh workunit to test the rbd-mirror daemon in journal mode'}], 'name': 'dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing-squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing-squid', 'sha1': '9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 5306, 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220', 'sleep_before_teardown': 0, 'subset': '21/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': '9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2', 'targets': {'smithi106.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEhs3TLZ8P9r3DOr+8IGGUHcnfAVbx7TqpskSVSFgCjPGwIhD09SIBtp+Z7rjLdnhz1N5Qf4WDMEJUB4B73G1wU=', 'smithi142.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAA1toN6R9IHXeOAxGgCT6AY7lk0P4OC4lEcl3mqalcJ3g8GXSNmbOrbouPGsGsi9R87RjctY7iw/lylaW12BKM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'journal', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '1ae7ad82388e92a475afff437d49054826c019a1', 'timestamp': '2024-05-07_23:02:08', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'}
2024-05-08T01:13:18.366 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data
2024-05-08T01:13:18.395 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data
2024-05-08T01:13:18.401 DEBUG:teuthology.orchestra.run.smithi106:> sudo install -d -m0777 -- /var/run/ceph
2024-05-08T01:13:18.443 DEBUG:teuthology.orchestra.run.smithi142:> sudo install -d -m0777 -- /var/run/ceph
2024-05-08T01:13:18.456 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:18.456 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/scratch_devs of=/dev/stdout
2024-05-08T01:13:18.502 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4']
2024-05-08T01:13:18.503 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_1
2024-05-08T01:13:18.550 INFO:teuthology.orchestra.run.smithi106.stdout:  File: /dev/vg_nvme/lv_1 -> ../dm-0
2024-05-08T01:13:18.551 INFO:teuthology.orchestra.run.smithi106.stdout:  Size: 7         	Blocks: 0          IO Block: 4096   symbolic link
2024-05-08T01:13:18.551 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d	Inode: 881         Links: 1
2024-05-08T01:13:18.551 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
2024-05-08T01:13:18.551 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-05-08 01:07:35.953068784 +0000
2024-05-08T01:13:18.551 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-05-08 01:07:35.681071210 +0000
2024-05-08T01:13:18.551 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-05-08 01:07:35.681071210 +0000
2024-05-08T01:13:18.551 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: -
2024-05-08T01:13:18.552 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1
2024-05-08T01:13:18.605 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in
2024-05-08T01:13:18.605 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out
2024-05-08T01:13:18.605 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000387741 s, 1.3 MB/s
2024-05-08T01:13:18.607 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1
2024-05-08T01:13:18.655 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_2
2024-05-08T01:13:18.702 INFO:teuthology.orchestra.run.smithi106.stdout:  File: /dev/vg_nvme/lv_2 -> ../dm-1
2024-05-08T01:13:18.702 INFO:teuthology.orchestra.run.smithi106.stdout:  Size: 7         	Blocks: 0          IO Block: 4096   symbolic link
2024-05-08T01:13:18.702 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d	Inode: 892         Links: 1
2024-05-08T01:13:18.702 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
2024-05-08T01:13:18.702 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-05-08 01:07:36.193066642 +0000
2024-05-08T01:13:18.702 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-05-08 01:07:36.189066679 +0000
2024-05-08T01:13:18.703 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-05-08 01:07:36.189066679 +0000
2024-05-08T01:13:18.703 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: -
2024-05-08T01:13:18.703 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1
2024-05-08T01:13:18.756 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in
2024-05-08T01:13:18.756 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out
2024-05-08T01:13:18.756 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.00042221 s, 1.2 MB/s
2024-05-08T01:13:18.758 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2
2024-05-08T01:13:18.807 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_3
2024-05-08T01:13:18.854 INFO:teuthology.orchestra.run.smithi106.stdout:  File: /dev/vg_nvme/lv_3 -> ../dm-2
2024-05-08T01:13:18.854 INFO:teuthology.orchestra.run.smithi106.stdout:  Size: 7         	Blocks: 0          IO Block: 4096   symbolic link
2024-05-08T01:13:18.854 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d	Inode: 912         Links: 1
2024-05-08T01:13:18.854 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
2024-05-08T01:13:18.855 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-05-08 01:07:36.925060112 +0000
2024-05-08T01:13:18.855 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-05-08 01:07:36.685062254 +0000
2024-05-08T01:13:18.855 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-05-08 01:07:36.685062254 +0000
2024-05-08T01:13:18.855 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: -
2024-05-08T01:13:18.855 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1
2024-05-08T01:13:18.908 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in
2024-05-08T01:13:18.908 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out
2024-05-08T01:13:18.908 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000405504 s, 1.3 MB/s
2024-05-08T01:13:18.909 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3
2024-05-08T01:13:18.960 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_4
2024-05-08T01:13:19.010 INFO:teuthology.orchestra.run.smithi106.stdout:  File: /dev/vg_nvme/lv_4 -> ../dm-3
2024-05-08T01:13:19.010 INFO:teuthology.orchestra.run.smithi106.stdout:  Size: 7         	Blocks: 0          IO Block: 4096   symbolic link
2024-05-08T01:13:19.010 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d	Inode: 933         Links: 1
2024-05-08T01:13:19.010 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
2024-05-08T01:13:19.010 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-05-08 01:07:37.453055403 +0000
2024-05-08T01:13:19.010 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-05-08 01:07:37.185057794 +0000
2024-05-08T01:13:19.010 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-05-08 01:07:37.185057794 +0000
2024-05-08T01:13:19.011 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: -
2024-05-08T01:13:19.011 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1
2024-05-08T01:13:19.065 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in
2024-05-08T01:13:19.066 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out
2024-05-08T01:13:19.066 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.00040971 s, 1.2 MB/s
2024-05-08T01:13:19.067 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4
2024-05-08T01:13:19.116 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}
2024-05-08T01:13:19.116 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi106.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}}
2024-05-08T01:13:19.116 INFO:tasks.ceph:Generating config...
2024-05-08T01:13:19.117 INFO:tasks.ceph:[client] debug rbd = 20
2024-05-08T01:13:19.118 INFO:tasks.ceph:[client] debug rbd_mirror = 15
2024-05-08T01:13:19.118 INFO:tasks.ceph:[client] log to stderr = False
2024-05-08T01:13:19.118 INFO:tasks.ceph:[client] rbd default features = 125
2024-05-08T01:13:19.118 INFO:tasks.ceph:[client] rbd mirror image policy type = none
2024-05-08T01:13:19.118 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.118 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.118 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.118 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.118 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.118 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.118 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.118 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.119 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.119 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.119 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.119 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.119 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.119 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.119 INFO:tasks.ceph:[global] mon client directed command retry = 5
2024-05-08T01:13:19.119 INFO:tasks.ceph:[global] mon warn on pool no app = False
2024-05-08T01:13:19.119 INFO:tasks.ceph:[global] ms inject socket failures = 5000
2024-05-08T01:13:19.119 INFO:tasks.ceph:[mgr] debug mgr = 20
2024-05-08T01:13:19.119 INFO:tasks.ceph:[mgr] debug ms = 1
2024-05-08T01:13:19.119 INFO:tasks.ceph:[mon] debug mon = 20
2024-05-08T01:13:19.119 INFO:tasks.ceph:[mon] debug ms = 1
2024-05-08T01:13:19.119 INFO:tasks.ceph:[mon] debug paxos = 20
2024-05-08T01:13:19.120 INFO:tasks.ceph:[osd] bluefs allocator = hybrid
2024-05-08T01:13:19.120 INFO:tasks.ceph:[osd] bluestore allocator = hybrid
2024-05-08T01:13:19.120 INFO:tasks.ceph:[osd] bluestore block size = 96636764160
2024-05-08T01:13:19.120 INFO:tasks.ceph:[osd] bluestore fsck on mount = True
2024-05-08T01:13:19.120 INFO:tasks.ceph:[osd] debug bluefs = 1/20
2024-05-08T01:13:19.120 INFO:tasks.ceph:[osd] debug bluestore = 1/20
2024-05-08T01:13:19.120 INFO:tasks.ceph:[osd] debug ms = 1
2024-05-08T01:13:19.120 INFO:tasks.ceph:[osd] debug osd = 20
2024-05-08T01:13:19.120 INFO:tasks.ceph:[osd] debug rocksdb = 4/10
2024-05-08T01:13:19.120 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85
2024-05-08T01:13:19.120 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9
2024-05-08T01:13:19.120 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8
2024-05-08T01:13:19.120 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95
2024-05-08T01:13:19.120 INFO:tasks.ceph:[osd] osd objectstore = bluestore
2024-05-08T01:13:19.121 INFO:tasks.ceph:Setting up cluster1.mon.a...
2024-05-08T01:13:19.121 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring
2024-05-08T01:13:19.189 INFO:teuthology.orchestra.run.smithi106.stdout:creating /etc/ceph/cluster1.keyring
2024-05-08T01:13:19.204 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring
2024-05-08T01:13:19.280 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0644 /etc/ceph/cluster1.keyring
2024-05-08T01:13:19.338 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.106')]
2024-05-08T01:13:19.339 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.106', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}}
2024-05-08T01:13:19.339 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:19.340 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf
2024-05-08T01:13:19.386 DEBUG:teuthology.orchestra.run.smithi106:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.106 --print /home/ubuntu/cephtest/cluster1.monmap
2024-05-08T01:13:19.464 INFO:teuthology.orchestra.run.smithi106.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap
2024-05-08T01:13:19.464 INFO:teuthology.orchestra.run.smithi106.stdout:monmaptool: generated fsid 3284bad2-5fe9-40c3-bba2-9a50da63f434
2024-05-08T01:13:19.464 INFO:teuthology.orchestra.run.smithi106.stdout:setting min_mon_release = quincy
2024-05-08T01:13:19.464 INFO:teuthology.orchestra.run.smithi106.stdout:epoch 0
2024-05-08T01:13:19.464 INFO:teuthology.orchestra.run.smithi106.stdout:fsid 3284bad2-5fe9-40c3-bba2-9a50da63f434
2024-05-08T01:13:19.465 INFO:teuthology.orchestra.run.smithi106.stdout:last_changed 2024-05-08T01:13:19.486465+0000
2024-05-08T01:13:19.465 INFO:teuthology.orchestra.run.smithi106.stdout:created 2024-05-08T01:13:19.486465+0000
2024-05-08T01:13:19.465 INFO:teuthology.orchestra.run.smithi106.stdout:min_mon_release 17 (quincy)
2024-05-08T01:13:19.465 INFO:teuthology.orchestra.run.smithi106.stdout:election_strategy: 1
2024-05-08T01:13:19.465 INFO:teuthology.orchestra.run.smithi106.stdout:0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a
2024-05-08T01:13:19.465 INFO:teuthology.orchestra.run.smithi106.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors)
2024-05-08T01:13:19.466 DEBUG:teuthology.orchestra.run.smithi106:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf
2024-05-08T01:13:19.514 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 3284bad2-5fe9-40c3-bba2-9a50da63f434...
2024-05-08T01:13:19.515 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null
2024-05-08T01:13:19.559 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null
2024-05-08T01:13:19.584 INFO:teuthology.orchestra.run.smithi106.stdout:[global]
2024-05-08T01:13:19.585 INFO:teuthology.orchestra.run.smithi106.stdout:	chdir = ""
2024-05-08T01:13:19.585 INFO:teuthology.orchestra.run.smithi106.stdout:	pid file = /var/run/ceph/$cluster-$name.pid
2024-05-08T01:13:19.585 INFO:teuthology.orchestra.run.smithi106.stdout:	auth supported = cephx
2024-05-08T01:13:19.585 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.585 INFO:teuthology.orchestra.run.smithi106.stdout:	filestore xattr use omap = true
2024-05-08T01:13:19.585 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.585 INFO:teuthology.orchestra.run.smithi106.stdout:	mon clock drift allowed = 1.000
2024-05-08T01:13:19.585 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.585 INFO:teuthology.orchestra.run.smithi106.stdout:	osd crush chooseleaf type = 0
2024-05-08T01:13:19.585 INFO:teuthology.orchestra.run.smithi106.stdout:	auth debug = true
2024-05-08T01:13:19.585 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.585 INFO:teuthology.orchestra.run.smithi106.stdout:	ms die on old message = true
2024-05-08T01:13:19.585 INFO:teuthology.orchestra.run.smithi106.stdout:	ms die on bug = true
2024-05-08T01:13:19.585 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.585 INFO:teuthology.orchestra.run.smithi106.stdout:	mon max pg per osd = 10000	# >= luminous
2024-05-08T01:13:19.586 INFO:teuthology.orchestra.run.smithi106.stdout:	mon pg warn max object skew = 0
2024-05-08T01:13:19.586 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.586 INFO:teuthology.orchestra.run.smithi106.stdout:	# disable pg_autoscaler by default for new pools
2024-05-08T01:13:19.586 INFO:teuthology.orchestra.run.smithi106.stdout:	osd_pool_default_pg_autoscale_mode = off
2024-05-08T01:13:19.586 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.586 INFO:teuthology.orchestra.run.smithi106.stdout:	osd pool default size = 2
2024-05-08T01:13:19.586 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.586 INFO:teuthology.orchestra.run.smithi106.stdout:	mon osd allow primary affinity = true
2024-05-08T01:13:19.586 INFO:teuthology.orchestra.run.smithi106.stdout:	mon osd allow pg remap = true
2024-05-08T01:13:19.586 INFO:teuthology.orchestra.run.smithi106.stdout:	mon warn on legacy crush tunables = false
2024-05-08T01:13:19.586 INFO:teuthology.orchestra.run.smithi106.stdout:	mon warn on crush straw calc version zero = false
2024-05-08T01:13:19.586 INFO:teuthology.orchestra.run.smithi106.stdout:	mon warn on no sortbitwise = false
2024-05-08T01:13:19.586 INFO:teuthology.orchestra.run.smithi106.stdout:	mon warn on osd down out interval zero = false
2024-05-08T01:13:19.586 INFO:teuthology.orchestra.run.smithi106.stdout:	mon warn on too few osds = false
2024-05-08T01:13:19.586 INFO:teuthology.orchestra.run.smithi106.stdout:	mon_warn_on_pool_pg_num_not_power_of_two = false
2024-05-08T01:13:19.587 INFO:teuthology.orchestra.run.smithi106.stdout:	mon_warn_on_pool_no_redundancy = false
2024-05-08T01:13:19.587 INFO:teuthology.orchestra.run.smithi106.stdout:	mon_allow_pool_size_one = true
2024-05-08T01:13:19.587 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.587 INFO:teuthology.orchestra.run.smithi106.stdout:	osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd
2024-05-08T01:13:19.587 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.587 INFO:teuthology.orchestra.run.smithi106.stdout:	osd default data pool replay window = 5
2024-05-08T01:13:19.587 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.587 INFO:teuthology.orchestra.run.smithi106.stdout:	mon allow pool delete = true
2024-05-08T01:13:19.587 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.587 INFO:teuthology.orchestra.run.smithi106.stdout:	mon cluster log file level = debug
2024-05-08T01:13:19.587 INFO:teuthology.orchestra.run.smithi106.stdout:	debug asserts on shutdown = true
2024-05-08T01:13:19.587 INFO:teuthology.orchestra.run.smithi106.stdout:	mon health detail to clog = false
2024-05-08T01:13:19.587 INFO:teuthology.orchestra.run.smithi106.stdout:	mon host = 172.21.15.106
2024-05-08T01:13:19.587 INFO:teuthology.orchestra.run.smithi106.stdout:	mon client directed command retry = 5
2024-05-08T01:13:19.587 INFO:teuthology.orchestra.run.smithi106.stdout:	mon warn on pool no app = False
2024-05-08T01:13:19.588 INFO:teuthology.orchestra.run.smithi106.stdout:	ms inject socket failures = 5000
2024-05-08T01:13:19.588 INFO:teuthology.orchestra.run.smithi106.stdout:	fsid = 3284bad2-5fe9-40c3-bba2-9a50da63f434
2024-05-08T01:13:19.588 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.588 INFO:teuthology.orchestra.run.smithi106.stdout:[osd]
2024-05-08T01:13:19.588 INFO:teuthology.orchestra.run.smithi106.stdout:	osd journal size = 100
2024-05-08T01:13:19.588 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.588 INFO:teuthology.orchestra.run.smithi106.stdout:	osd scrub load threshold = 5.0
2024-05-08T01:13:19.588 INFO:teuthology.orchestra.run.smithi106.stdout:	osd scrub max interval = 600
2024-05-08T01:13:19.588 INFO:teuthology.orchestra.run.smithi106.stdout:	osd mclock profile = high_recovery_ops
2024-05-08T01:13:19.588 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.588 INFO:teuthology.orchestra.run.smithi106.stdout:	osd recover clone overlap = true
2024-05-08T01:13:19.588 INFO:teuthology.orchestra.run.smithi106.stdout:	osd recovery max chunk = 1048576
2024-05-08T01:13:19.588 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.588 INFO:teuthology.orchestra.run.smithi106.stdout:	osd debug shutdown = true
2024-05-08T01:13:19.588 INFO:teuthology.orchestra.run.smithi106.stdout:	osd debug op order = true
2024-05-08T01:13:19.589 INFO:teuthology.orchestra.run.smithi106.stdout:	osd debug verify stray on activate = true
2024-05-08T01:13:19.589 INFO:teuthology.orchestra.run.smithi106.stdout:	osd debug trim objects = true
2024-05-08T01:13:19.589 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.589 INFO:teuthology.orchestra.run.smithi106.stdout:	osd open classes on start = true
2024-05-08T01:13:19.589 INFO:teuthology.orchestra.run.smithi106.stdout:	osd debug pg log writeout = true
2024-05-08T01:13:19.589 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.589 INFO:teuthology.orchestra.run.smithi106.stdout:	osd deep scrub update digest min age = 30
2024-05-08T01:13:19.589 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.589 INFO:teuthology.orchestra.run.smithi106.stdout:	osd map max advance = 10
2024-05-08T01:13:19.589 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.589 INFO:teuthology.orchestra.run.smithi106.stdout:	journal zero on create = true
2024-05-08T01:13:19.589 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.590 INFO:teuthology.orchestra.run.smithi106.stdout:	filestore ondisk finisher threads = 3
2024-05-08T01:13:19.590 INFO:teuthology.orchestra.run.smithi106.stdout:	filestore apply finisher threads = 3
2024-05-08T01:13:19.590 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.590 INFO:teuthology.orchestra.run.smithi106.stdout:	bdev debug aio = true
2024-05-08T01:13:19.590 INFO:teuthology.orchestra.run.smithi106.stdout:	osd debug misdirected ops = true
2024-05-08T01:13:19.590 INFO:teuthology.orchestra.run.smithi106.stdout:	bluefs allocator = hybrid
2024-05-08T01:13:19.590 INFO:teuthology.orchestra.run.smithi106.stdout:	bluestore allocator = hybrid
2024-05-08T01:13:19.590 INFO:teuthology.orchestra.run.smithi106.stdout:	bluestore block size = 96636764160
2024-05-08T01:13:19.590 INFO:teuthology.orchestra.run.smithi106.stdout:	bluestore fsck on mount = True
2024-05-08T01:13:19.590 INFO:teuthology.orchestra.run.smithi106.stdout:	debug bluefs = 1/20
2024-05-08T01:13:19.590 INFO:teuthology.orchestra.run.smithi106.stdout:	debug bluestore = 1/20
2024-05-08T01:13:19.590 INFO:teuthology.orchestra.run.smithi106.stdout:	debug ms = 1
2024-05-08T01:13:19.590 INFO:teuthology.orchestra.run.smithi106.stdout:	debug osd = 20
2024-05-08T01:13:19.590 INFO:teuthology.orchestra.run.smithi106.stdout:	debug rocksdb = 4/10
2024-05-08T01:13:19.590 INFO:teuthology.orchestra.run.smithi106.stdout:	mon osd backfillfull_ratio = 0.85
2024-05-08T01:13:19.591 INFO:teuthology.orchestra.run.smithi106.stdout:	mon osd full ratio = 0.9
2024-05-08T01:13:19.591 INFO:teuthology.orchestra.run.smithi106.stdout:	mon osd nearfull ratio = 0.8
2024-05-08T01:13:19.591 INFO:teuthology.orchestra.run.smithi106.stdout:	osd failsafe full ratio = 0.95
2024-05-08T01:13:19.591 INFO:teuthology.orchestra.run.smithi106.stdout:	osd objectstore = bluestore
2024-05-08T01:13:19.591 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.591 INFO:teuthology.orchestra.run.smithi106.stdout:[mgr]
2024-05-08T01:13:19.591 INFO:teuthology.orchestra.run.smithi106.stdout:	debug ms = 1
2024-05-08T01:13:19.591 INFO:teuthology.orchestra.run.smithi106.stdout:	debug mgr = 20
2024-05-08T01:13:19.591 INFO:teuthology.orchestra.run.smithi106.stdout:	debug mon = 20
2024-05-08T01:13:19.591 INFO:teuthology.orchestra.run.smithi106.stdout:	debug auth = 20
2024-05-08T01:13:19.591 INFO:teuthology.orchestra.run.smithi106.stdout:	mon reweight min pgs per osd = 4
2024-05-08T01:13:19.591 INFO:teuthology.orchestra.run.smithi106.stdout:	mon reweight min bytes per osd = 10
2024-05-08T01:13:19.591 INFO:teuthology.orchestra.run.smithi106.stdout:	mgr/telemetry/nag = false
2024-05-08T01:13:19.591 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.591 INFO:teuthology.orchestra.run.smithi106.stdout:[mon]
2024-05-08T01:13:19.592 INFO:teuthology.orchestra.run.smithi106.stdout:	debug ms = 1
2024-05-08T01:13:19.592 INFO:teuthology.orchestra.run.smithi106.stdout:	debug mon = 20
2024-05-08T01:13:19.592 INFO:teuthology.orchestra.run.smithi106.stdout:	debug paxos = 20
2024-05-08T01:13:19.592 INFO:teuthology.orchestra.run.smithi106.stdout:	debug auth = 20
2024-05-08T01:13:19.592 INFO:teuthology.orchestra.run.smithi106.stdout:	mon data avail warn = 5
2024-05-08T01:13:19.592 INFO:teuthology.orchestra.run.smithi106.stdout:	mon mgr mkfs grace = 240
2024-05-08T01:13:19.592 INFO:teuthology.orchestra.run.smithi106.stdout:	mon reweight min pgs per osd = 4
2024-05-08T01:13:19.592 INFO:teuthology.orchestra.run.smithi106.stdout:	mon osd reporter subtree level = osd
2024-05-08T01:13:19.592 INFO:teuthology.orchestra.run.smithi106.stdout:	mon osd prime pg temp = true
2024-05-08T01:13:19.592 INFO:teuthology.orchestra.run.smithi106.stdout:	mon reweight min bytes per osd = 10
2024-05-08T01:13:19.592 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.592 INFO:teuthology.orchestra.run.smithi106.stdout:	# rotate auth tickets quickly to exercise renewal paths
2024-05-08T01:13:19.592 INFO:teuthology.orchestra.run.smithi106.stdout:	auth mon ticket ttl = 660	# 11m
2024-05-08T01:13:19.592 INFO:teuthology.orchestra.run.smithi106.stdout:	auth service ticket ttl = 240	# 4m
2024-05-08T01:13:19.593 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.593 INFO:teuthology.orchestra.run.smithi106.stdout:	# don't complain about insecure global_id in the test suite
2024-05-08T01:13:19.593 INFO:teuthology.orchestra.run.smithi106.stdout:	mon_warn_on_insecure_global_id_reclaim = false
2024-05-08T01:13:19.593 INFO:teuthology.orchestra.run.smithi106.stdout:	mon_warn_on_insecure_global_id_reclaim_allowed = false
2024-05-08T01:13:19.593 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.593 INFO:teuthology.orchestra.run.smithi106.stdout:	# 1m isn't quite enough
2024-05-08T01:13:19.593 INFO:teuthology.orchestra.run.smithi106.stdout:	mon_down_mkfs_grace = 2m
2024-05-08T01:13:19.593 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.593 INFO:teuthology.orchestra.run.smithi106.stdout:	mon_warn_on_filestore_osds = false
2024-05-08T01:13:19.593 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:19.593 INFO:teuthology.orchestra.run.smithi106.stdout:[client]
2024-05-08T01:13:19.593 INFO:teuthology.orchestra.run.smithi106.stdout:	rgw cache enabled = true
2024-05-08T01:13:19.593 INFO:teuthology.orchestra.run.smithi106.stdout:	rgw enable ops log = true
2024-05-08T01:13:19.593 INFO:teuthology.orchestra.run.smithi106.stdout:	rgw enable usage log = true
2024-05-08T01:13:19.594 INFO:teuthology.orchestra.run.smithi106.stdout:	log file = /var/log/ceph/$cluster-$name.$pid.log
2024-05-08T01:13:19.594 INFO:teuthology.orchestra.run.smithi106.stdout:	admin socket = /var/run/ceph/$cluster-$name.$pid.asok
2024-05-08T01:13:19.594 INFO:teuthology.orchestra.run.smithi106.stdout:	debug rbd = 20
2024-05-08T01:13:19.594 INFO:teuthology.orchestra.run.smithi106.stdout:	debug rbd_mirror = 15
2024-05-08T01:13:19.594 INFO:teuthology.orchestra.run.smithi106.stdout:	log to stderr = False
2024-05-08T01:13:19.594 INFO:teuthology.orchestra.run.smithi106.stdout:	rbd default features = 125
2024-05-08T01:13:19.594 INFO:teuthology.orchestra.run.smithi106.stdout:	rbd mirror image policy type = none
2024-05-08T01:13:19.594 INFO:teuthology.orchestra.run.smithi106.stdout:[mon.a]
2024-05-08T01:13:19.594 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.0]
2024-05-08T01:13:19.594 INFO:teuthology.orchestra.run.smithi106.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.594 INFO:teuthology.orchestra.run.smithi106.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.594 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.1]
2024-05-08T01:13:19.594 INFO:teuthology.orchestra.run.smithi106.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.595 INFO:teuthology.orchestra.run.smithi106.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.595 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.2]
2024-05-08T01:13:19.595 INFO:teuthology.orchestra.run.smithi106.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.595 INFO:teuthology.orchestra.run.smithi106.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.595 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.3]
2024-05-08T01:13:19.595 INFO:teuthology.orchestra.run.smithi106.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.595 INFO:teuthology.orchestra.run.smithi106.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.595 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.4]
2024-05-08T01:13:19.595 INFO:teuthology.orchestra.run.smithi106.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.595 INFO:teuthology.orchestra.run.smithi106.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.595 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.5]
2024-05-08T01:13:19.595 INFO:teuthology.orchestra.run.smithi106.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.595 INFO:teuthology.orchestra.run.smithi106.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.595 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.6]
2024-05-08T01:13:19.595 INFO:teuthology.orchestra.run.smithi106.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.596 INFO:teuthology.orchestra.run.smithi106.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.596 INFO:teuthology.orchestra.run.smithi142.stdout:[global]
2024-05-08T01:13:19.596 INFO:teuthology.orchestra.run.smithi142.stdout:	chdir = ""
2024-05-08T01:13:19.596 INFO:teuthology.orchestra.run.smithi142.stdout:	pid file = /var/run/ceph/$cluster-$name.pid
2024-05-08T01:13:19.596 INFO:teuthology.orchestra.run.smithi142.stdout:	auth supported = cephx
2024-05-08T01:13:19.596 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.597 INFO:teuthology.orchestra.run.smithi142.stdout:	filestore xattr use omap = true
2024-05-08T01:13:19.597 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.597 INFO:teuthology.orchestra.run.smithi142.stdout:	mon clock drift allowed = 1.000
2024-05-08T01:13:19.597 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.597 INFO:teuthology.orchestra.run.smithi142.stdout:	osd crush chooseleaf type = 0
2024-05-08T01:13:19.597 INFO:teuthology.orchestra.run.smithi142.stdout:	auth debug = true
2024-05-08T01:13:19.597 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.597 INFO:teuthology.orchestra.run.smithi142.stdout:	ms die on old message = true
2024-05-08T01:13:19.597 INFO:teuthology.orchestra.run.smithi142.stdout:	ms die on bug = true
2024-05-08T01:13:19.597 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.597 INFO:teuthology.orchestra.run.smithi142.stdout:	mon max pg per osd = 10000	# >= luminous
2024-05-08T01:13:19.597 INFO:teuthology.orchestra.run.smithi142.stdout:	mon pg warn max object skew = 0
2024-05-08T01:13:19.597 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.597 INFO:teuthology.orchestra.run.smithi142.stdout:	# disable pg_autoscaler by default for new pools
2024-05-08T01:13:19.598 INFO:teuthology.orchestra.run.smithi142.stdout:	osd_pool_default_pg_autoscale_mode = off
2024-05-08T01:13:19.598 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.598 INFO:teuthology.orchestra.run.smithi142.stdout:	osd pool default size = 2
2024-05-08T01:13:19.598 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.598 INFO:teuthology.orchestra.run.smithi142.stdout:	mon osd allow primary affinity = true
2024-05-08T01:13:19.598 INFO:teuthology.orchestra.run.smithi142.stdout:	mon osd allow pg remap = true
2024-05-08T01:13:19.598 INFO:teuthology.orchestra.run.smithi142.stdout:	mon warn on legacy crush tunables = false
2024-05-08T01:13:19.598 INFO:teuthology.orchestra.run.smithi142.stdout:	mon warn on crush straw calc version zero = false
2024-05-08T01:13:19.598 INFO:teuthology.orchestra.run.smithi142.stdout:	mon warn on no sortbitwise = false
2024-05-08T01:13:19.598 INFO:teuthology.orchestra.run.smithi142.stdout:	mon warn on osd down out interval zero = false
2024-05-08T01:13:19.598 INFO:teuthology.orchestra.run.smithi142.stdout:	mon warn on too few osds = false
2024-05-08T01:13:19.598 INFO:teuthology.orchestra.run.smithi142.stdout:	mon_warn_on_pool_pg_num_not_power_of_two = false
2024-05-08T01:13:19.598 INFO:teuthology.orchestra.run.smithi142.stdout:	mon_warn_on_pool_no_redundancy = false
2024-05-08T01:13:19.598 INFO:teuthology.orchestra.run.smithi142.stdout:	mon_allow_pool_size_one = true
2024-05-08T01:13:19.599 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.599 INFO:teuthology.orchestra.run.smithi142.stdout:	osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd
2024-05-08T01:13:19.599 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.599 INFO:teuthology.orchestra.run.smithi142.stdout:	osd default data pool replay window = 5
2024-05-08T01:13:19.599 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.599 INFO:teuthology.orchestra.run.smithi142.stdout:	mon allow pool delete = true
2024-05-08T01:13:19.599 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.599 INFO:teuthology.orchestra.run.smithi142.stdout:	mon cluster log file level = debug
2024-05-08T01:13:19.599 INFO:teuthology.orchestra.run.smithi142.stdout:	debug asserts on shutdown = true
2024-05-08T01:13:19.599 INFO:teuthology.orchestra.run.smithi142.stdout:	mon health detail to clog = false
2024-05-08T01:13:19.599 INFO:teuthology.orchestra.run.smithi142.stdout:	mon host = 172.21.15.106
2024-05-08T01:13:19.599 INFO:teuthology.orchestra.run.smithi142.stdout:	mon client directed command retry = 5
2024-05-08T01:13:19.599 INFO:teuthology.orchestra.run.smithi142.stdout:	mon warn on pool no app = False
2024-05-08T01:13:19.599 INFO:teuthology.orchestra.run.smithi142.stdout:	ms inject socket failures = 5000
2024-05-08T01:13:19.600 INFO:teuthology.orchestra.run.smithi142.stdout:	fsid = 3284bad2-5fe9-40c3-bba2-9a50da63f434
2024-05-08T01:13:19.600 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.600 INFO:teuthology.orchestra.run.smithi142.stdout:[osd]
2024-05-08T01:13:19.600 INFO:teuthology.orchestra.run.smithi142.stdout:	osd journal size = 100
2024-05-08T01:13:19.600 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.600 INFO:teuthology.orchestra.run.smithi142.stdout:	osd scrub load threshold = 5.0
2024-05-08T01:13:19.600 INFO:teuthology.orchestra.run.smithi142.stdout:	osd scrub max interval = 600
2024-05-08T01:13:19.600 INFO:teuthology.orchestra.run.smithi142.stdout:	osd mclock profile = high_recovery_ops
2024-05-08T01:13:19.600 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.600 INFO:teuthology.orchestra.run.smithi142.stdout:	osd recover clone overlap = true
2024-05-08T01:13:19.600 INFO:teuthology.orchestra.run.smithi142.stdout:	osd recovery max chunk = 1048576
2024-05-08T01:13:19.600 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.600 INFO:teuthology.orchestra.run.smithi142.stdout:	osd debug shutdown = true
2024-05-08T01:13:19.600 INFO:teuthology.orchestra.run.smithi142.stdout:	osd debug op order = true
2024-05-08T01:13:19.600 INFO:teuthology.orchestra.run.smithi142.stdout:	osd debug verify stray on activate = true
2024-05-08T01:13:19.600 INFO:teuthology.orchestra.run.smithi142.stdout:	osd debug trim objects = true
2024-05-08T01:13:19.601 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.601 INFO:teuthology.orchestra.run.smithi142.stdout:	osd open classes on start = true
2024-05-08T01:13:19.601 INFO:teuthology.orchestra.run.smithi142.stdout:	osd debug pg log writeout = true
2024-05-08T01:13:19.601 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.601 INFO:teuthology.orchestra.run.smithi142.stdout:	osd deep scrub update digest min age = 30
2024-05-08T01:13:19.601 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.601 INFO:teuthology.orchestra.run.smithi142.stdout:	osd map max advance = 10
2024-05-08T01:13:19.601 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.601 INFO:teuthology.orchestra.run.smithi142.stdout:	journal zero on create = true
2024-05-08T01:13:19.601 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.601 INFO:teuthology.orchestra.run.smithi142.stdout:	filestore ondisk finisher threads = 3
2024-05-08T01:13:19.601 INFO:teuthology.orchestra.run.smithi142.stdout:	filestore apply finisher threads = 3
2024-05-08T01:13:19.601 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.601 INFO:teuthology.orchestra.run.smithi142.stdout:	bdev debug aio = true
2024-05-08T01:13:19.601 INFO:teuthology.orchestra.run.smithi142.stdout:	osd debug misdirected ops = true
2024-05-08T01:13:19.602 INFO:teuthology.orchestra.run.smithi142.stdout:	bluefs allocator = hybrid
2024-05-08T01:13:19.602 INFO:teuthology.orchestra.run.smithi142.stdout:	bluestore allocator = hybrid
2024-05-08T01:13:19.602 INFO:teuthology.orchestra.run.smithi142.stdout:	bluestore block size = 96636764160
2024-05-08T01:13:19.602 INFO:teuthology.orchestra.run.smithi142.stdout:	bluestore fsck on mount = True
2024-05-08T01:13:19.602 INFO:teuthology.orchestra.run.smithi142.stdout:	debug bluefs = 1/20
2024-05-08T01:13:19.602 INFO:teuthology.orchestra.run.smithi142.stdout:	debug bluestore = 1/20
2024-05-08T01:13:19.602 INFO:teuthology.orchestra.run.smithi142.stdout:	debug ms = 1
2024-05-08T01:13:19.602 INFO:teuthology.orchestra.run.smithi142.stdout:	debug osd = 20
2024-05-08T01:13:19.602 INFO:teuthology.orchestra.run.smithi142.stdout:	debug rocksdb = 4/10
2024-05-08T01:13:19.602 INFO:teuthology.orchestra.run.smithi142.stdout:	mon osd backfillfull_ratio = 0.85
2024-05-08T01:13:19.602 INFO:teuthology.orchestra.run.smithi142.stdout:	mon osd full ratio = 0.9
2024-05-08T01:13:19.602 INFO:teuthology.orchestra.run.smithi142.stdout:	mon osd nearfull ratio = 0.8
2024-05-08T01:13:19.603 INFO:teuthology.orchestra.run.smithi142.stdout:	osd failsafe full ratio = 0.95
2024-05-08T01:13:19.603 INFO:teuthology.orchestra.run.smithi142.stdout:	osd objectstore = bluestore
2024-05-08T01:13:19.603 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.603 INFO:teuthology.orchestra.run.smithi142.stdout:[mgr]
2024-05-08T01:13:19.603 INFO:teuthology.orchestra.run.smithi142.stdout:	debug ms = 1
2024-05-08T01:13:19.603 INFO:teuthology.orchestra.run.smithi142.stdout:	debug mgr = 20
2024-05-08T01:13:19.603 INFO:teuthology.orchestra.run.smithi142.stdout:	debug mon = 20
2024-05-08T01:13:19.603 INFO:teuthology.orchestra.run.smithi142.stdout:	debug auth = 20
2024-05-08T01:13:19.603 INFO:teuthology.orchestra.run.smithi142.stdout:	mon reweight min pgs per osd = 4
2024-05-08T01:13:19.603 INFO:teuthology.orchestra.run.smithi142.stdout:	mon reweight min bytes per osd = 10
2024-05-08T01:13:19.603 INFO:teuthology.orchestra.run.smithi142.stdout:	mgr/telemetry/nag = false
2024-05-08T01:13:19.603 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.603 INFO:teuthology.orchestra.run.smithi142.stdout:[mon]
2024-05-08T01:13:19.603 INFO:teuthology.orchestra.run.smithi142.stdout:	debug ms = 1
2024-05-08T01:13:19.604 INFO:teuthology.orchestra.run.smithi142.stdout:	debug mon = 20
2024-05-08T01:13:19.604 INFO:teuthology.orchestra.run.smithi142.stdout:	debug paxos = 20
2024-05-08T01:13:19.604 INFO:teuthology.orchestra.run.smithi142.stdout:	debug auth = 20
2024-05-08T01:13:19.604 INFO:teuthology.orchestra.run.smithi142.stdout:	mon data avail warn = 5
2024-05-08T01:13:19.604 INFO:teuthology.orchestra.run.smithi142.stdout:	mon mgr mkfs grace = 240
2024-05-08T01:13:19.604 INFO:teuthology.orchestra.run.smithi142.stdout:	mon reweight min pgs per osd = 4
2024-05-08T01:13:19.604 INFO:teuthology.orchestra.run.smithi142.stdout:	mon osd reporter subtree level = osd
2024-05-08T01:13:19.604 INFO:teuthology.orchestra.run.smithi142.stdout:	mon osd prime pg temp = true
2024-05-08T01:13:19.604 INFO:teuthology.orchestra.run.smithi142.stdout:	mon reweight min bytes per osd = 10
2024-05-08T01:13:19.604 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.604 INFO:teuthology.orchestra.run.smithi142.stdout:	# rotate auth tickets quickly to exercise renewal paths
2024-05-08T01:13:19.604 INFO:teuthology.orchestra.run.smithi142.stdout:	auth mon ticket ttl = 660	# 11m
2024-05-08T01:13:19.604 INFO:teuthology.orchestra.run.smithi142.stdout:	auth service ticket ttl = 240	# 4m
2024-05-08T01:13:19.604 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.604 INFO:teuthology.orchestra.run.smithi142.stdout:	# don't complain about insecure global_id in the test suite
2024-05-08T01:13:19.605 INFO:teuthology.orchestra.run.smithi142.stdout:	mon_warn_on_insecure_global_id_reclaim = false
2024-05-08T01:13:19.605 INFO:teuthology.orchestra.run.smithi142.stdout:	mon_warn_on_insecure_global_id_reclaim_allowed = false
2024-05-08T01:13:19.605 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.605 INFO:teuthology.orchestra.run.smithi142.stdout:	# 1m isn't quite enough
2024-05-08T01:13:19.605 INFO:teuthology.orchestra.run.smithi142.stdout:	mon_down_mkfs_grace = 2m
2024-05-08T01:13:19.605 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.605 INFO:teuthology.orchestra.run.smithi142.stdout:	mon_warn_on_filestore_osds = false
2024-05-08T01:13:19.605 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:19.605 INFO:teuthology.orchestra.run.smithi142.stdout:[client]
2024-05-08T01:13:19.605 INFO:teuthology.orchestra.run.smithi142.stdout:	rgw cache enabled = true
2024-05-08T01:13:19.605 INFO:teuthology.orchestra.run.smithi142.stdout:	rgw enable ops log = true
2024-05-08T01:13:19.605 INFO:teuthology.orchestra.run.smithi142.stdout:	rgw enable usage log = true
2024-05-08T01:13:19.605 INFO:teuthology.orchestra.run.smithi142.stdout:	log file = /var/log/ceph/$cluster-$name.$pid.log
2024-05-08T01:13:19.605 INFO:teuthology.orchestra.run.smithi142.stdout:	admin socket = /var/run/ceph/$cluster-$name.$pid.asok
2024-05-08T01:13:19.605 INFO:teuthology.orchestra.run.smithi142.stdout:	debug rbd = 20
2024-05-08T01:13:19.606 INFO:teuthology.orchestra.run.smithi142.stdout:	debug rbd_mirror = 15
2024-05-08T01:13:19.606 INFO:teuthology.orchestra.run.smithi142.stdout:	log to stderr = False
2024-05-08T01:13:19.606 INFO:teuthology.orchestra.run.smithi142.stdout:	rbd default features = 125
2024-05-08T01:13:19.606 INFO:teuthology.orchestra.run.smithi142.stdout:	rbd mirror image policy type = none
2024-05-08T01:13:19.606 INFO:teuthology.orchestra.run.smithi142.stdout:[mon.a]
2024-05-08T01:13:19.606 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.0]
2024-05-08T01:13:19.606 INFO:teuthology.orchestra.run.smithi142.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.606 INFO:teuthology.orchestra.run.smithi142.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.606 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.1]
2024-05-08T01:13:19.606 INFO:teuthology.orchestra.run.smithi142.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.606 INFO:teuthology.orchestra.run.smithi142.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.606 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.2]
2024-05-08T01:13:19.606 INFO:teuthology.orchestra.run.smithi142.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.606 INFO:teuthology.orchestra.run.smithi142.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.606 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.3]
2024-05-08T01:13:19.607 INFO:teuthology.orchestra.run.smithi142.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.607 INFO:teuthology.orchestra.run.smithi142.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.607 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.4]
2024-05-08T01:13:19.607 INFO:teuthology.orchestra.run.smithi142.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.607 INFO:teuthology.orchestra.run.smithi142.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.607 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.5]
2024-05-08T01:13:19.607 INFO:teuthology.orchestra.run.smithi142.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.607 INFO:teuthology.orchestra.run.smithi142.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.607 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.6]
2024-05-08T01:13:19.607 INFO:teuthology.orchestra.run.smithi142.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:19.607 INFO:teuthology.orchestra.run.smithi142.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:19.609 INFO:tasks.ceph:Creating admin key on cluster1.mon.a...
2024-05-08T01:13:19.609 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring
2024-05-08T01:13:19.697 INFO:tasks.ceph:Copying monmap to all nodes...
2024-05-08T01:13:19.698 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:19.698 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout
2024-05-08T01:13:19.703 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:19.703 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout
2024-05-08T01:13:19.749 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi106.front.sepia.ceph.com
2024-05-08T01:13:19.749 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:19.749 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/cluster1.keyring
2024-05-08T01:13:19.749 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0644 /etc/ceph/cluster1.keyring
2024-05-08T01:13:19.812 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:19.812 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/home/ubuntu/cephtest/cluster1.monmap
2024-05-08T01:13:19.861 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi142.front.sepia.ceph.com
2024-05-08T01:13:19.862 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:19.862 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/cluster1.keyring
2024-05-08T01:13:19.862 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0644 /etc/ceph/cluster1.keyring
2024-05-08T01:13:19.884 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:19.884 DEBUG:teuthology.orchestra.run.smithi142:> dd of=/home/ubuntu/cephtest/cluster1.monmap
2024-05-08T01:13:19.932 INFO:tasks.ceph:Setting up mon nodes...
2024-05-08T01:13:19.932 INFO:tasks.ceph:Setting up mgr nodes...
2024-05-08T01:13:19.932 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring
2024-05-08T01:13:19.975 INFO:teuthology.orchestra.run.smithi106.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring
2024-05-08T01:13:19.977 INFO:tasks.ceph:Setting up mds nodes...
2024-05-08T01:13:19.978 INFO:tasks.ceph_client:Setting up client nodes...
2024-05-08T01:13:19.978 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring
2024-05-08T01:13:20.053 INFO:teuthology.orchestra.run.smithi106.stdout:creating /etc/ceph/cluster1.client.0.keyring
2024-05-08T01:13:20.064 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring
2024-05-08T01:13:20.100 INFO:teuthology.orchestra.run.smithi142.stdout:creating /etc/ceph/cluster1.client.mirror.keyring
2024-05-08T01:13:20.130 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring
2024-05-08T01:13:20.203 INFO:teuthology.orchestra.run.smithi142.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring
2024-05-08T01:13:20.213 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring
2024-05-08T01:13:20.285 INFO:teuthology.orchestra.run.smithi142.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring
2024-05-08T01:13:20.295 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring
2024-05-08T01:13:20.369 INFO:teuthology.orchestra.run.smithi142.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring
2024-05-08T01:13:20.378 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring
2024-05-08T01:13:20.454 INFO:teuthology.orchestra.run.smithi142.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring
2024-05-08T01:13:20.464 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring
2024-05-08T01:13:20.538 INFO:teuthology.orchestra.run.smithi142.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring
2024-05-08T01:13:20.547 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring
2024-05-08T01:13:20.620 INFO:teuthology.orchestra.run.smithi142.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring
2024-05-08T01:13:20.630 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring
2024-05-08T01:13:20.698 INFO:teuthology.orchestra.run.smithi142.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring
2024-05-08T01:13:20.707 INFO:tasks.ceph:Running mkfs on osd nodes...
2024-05-08T01:13:20.708 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi106.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}}
2024-05-08T01:13:20.708 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0
2024-05-08T01:13:20.722 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}
2024-05-08T01:13:20.722 INFO:tasks.ceph:role: cluster1.osd.0
2024-05-08T01:13:20.723 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi106.front.sepia.ceph.com
2024-05-08T01:13:20.723 DEBUG:teuthology.orchestra.run.smithi106:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1
2024-05-08T01:13:20.786 INFO:teuthology.orchestra.run.smithi106.stdout:meta-data=/dev/vg_nvme/lv_1      isize=2048   agcount=4, agsize=5859072 blks
2024-05-08T01:13:20.786 INFO:teuthology.orchestra.run.smithi106.stdout:         =                       sectsz=512   attr=2, projid32bit=1
2024-05-08T01:13:20.786 INFO:teuthology.orchestra.run.smithi106.stdout:         =                       crc=1        finobt=1, sparse=1, rmapbt=0
2024-05-08T01:13:20.786 INFO:teuthology.orchestra.run.smithi106.stdout:         =                       reflink=1    bigtime=0 inobtcount=0
2024-05-08T01:13:20.786 INFO:teuthology.orchestra.run.smithi106.stdout:data     =                       bsize=4096   blocks=23436288, imaxpct=25
2024-05-08T01:13:20.787 INFO:teuthology.orchestra.run.smithi106.stdout:         =                       sunit=0      swidth=0 blks
2024-05-08T01:13:20.787 INFO:teuthology.orchestra.run.smithi106.stdout:naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
2024-05-08T01:13:20.787 INFO:teuthology.orchestra.run.smithi106.stdout:log      =internal log           bsize=4096   blocks=11443, version=2
2024-05-08T01:13:20.787 INFO:teuthology.orchestra.run.smithi106.stdout:         =                       sectsz=512   sunit=0 blks, lazy-count=1
2024-05-08T01:13:20.787 INFO:teuthology.orchestra.run.smithi106.stdout:realtime =none                   extsz=4096   blocks=0, rtextents=0
2024-05-08T01:13:21.242 INFO:teuthology.orchestra.run.smithi106.stdout:Discarding blocks...Done.
2024-05-08T01:13:21.244 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi106.front.sepia.ceph.com -o noatime
2024-05-08T01:13:21.244 DEBUG:teuthology.orchestra.run.smithi106:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0
2024-05-08T01:13:21.342 DEBUG:teuthology.orchestra.run.smithi106:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0
2024-05-08T01:13:21.353 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T01:13:21.354 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: /sbin/restorecon: command not found
2024-05-08T01:13:21.354 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1
2024-05-08T01:13:21.410 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}
2024-05-08T01:13:21.410 INFO:tasks.ceph:role: cluster1.osd.1
2024-05-08T01:13:21.410 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi106.front.sepia.ceph.com
2024-05-08T01:13:21.410 DEBUG:teuthology.orchestra.run.smithi106:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2
2024-05-08T01:13:21.474 INFO:teuthology.orchestra.run.smithi106.stdout:meta-data=/dev/vg_nvme/lv_2      isize=2048   agcount=4, agsize=5859072 blks
2024-05-08T01:13:21.475 INFO:teuthology.orchestra.run.smithi106.stdout:         =                       sectsz=512   attr=2, projid32bit=1
2024-05-08T01:13:21.475 INFO:teuthology.orchestra.run.smithi106.stdout:         =                       crc=1        finobt=1, sparse=1, rmapbt=0
2024-05-08T01:13:21.475 INFO:teuthology.orchestra.run.smithi106.stdout:         =                       reflink=1    bigtime=0 inobtcount=0
2024-05-08T01:13:21.475 INFO:teuthology.orchestra.run.smithi106.stdout:data     =                       bsize=4096   blocks=23436288, imaxpct=25
2024-05-08T01:13:21.475 INFO:teuthology.orchestra.run.smithi106.stdout:         =                       sunit=0      swidth=0 blks
2024-05-08T01:13:21.475 INFO:teuthology.orchestra.run.smithi106.stdout:naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
2024-05-08T01:13:21.476 INFO:teuthology.orchestra.run.smithi106.stdout:log      =internal log           bsize=4096   blocks=11443, version=2
2024-05-08T01:13:21.476 INFO:teuthology.orchestra.run.smithi106.stdout:         =                       sectsz=512   sunit=0 blks, lazy-count=1
2024-05-08T01:13:21.476 INFO:teuthology.orchestra.run.smithi106.stdout:realtime =none                   extsz=4096   blocks=0, rtextents=0
2024-05-08T01:13:21.942 INFO:teuthology.orchestra.run.smithi106.stdout:Discarding blocks...Done.
2024-05-08T01:13:21.944 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi106.front.sepia.ceph.com -o noatime
2024-05-08T01:13:21.945 DEBUG:teuthology.orchestra.run.smithi106:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1
2024-05-08T01:13:22.063 DEBUG:teuthology.orchestra.run.smithi106:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1
2024-05-08T01:13:22.075 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T01:13:22.075 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: /sbin/restorecon: command not found
2024-05-08T01:13:22.075 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2
2024-05-08T01:13:22.130 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}
2024-05-08T01:13:22.130 INFO:tasks.ceph:role: cluster1.osd.2
2024-05-08T01:13:22.130 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi106.front.sepia.ceph.com
2024-05-08T01:13:22.130 DEBUG:teuthology.orchestra.run.smithi106:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3
2024-05-08T01:13:22.197 INFO:teuthology.orchestra.run.smithi106.stdout:meta-data=/dev/vg_nvme/lv_3      isize=2048   agcount=4, agsize=5859072 blks
2024-05-08T01:13:22.197 INFO:teuthology.orchestra.run.smithi106.stdout:         =                       sectsz=512   attr=2, projid32bit=1
2024-05-08T01:13:22.197 INFO:teuthology.orchestra.run.smithi106.stdout:         =                       crc=1        finobt=1, sparse=1, rmapbt=0
2024-05-08T01:13:22.197 INFO:teuthology.orchestra.run.smithi106.stdout:         =                       reflink=1    bigtime=0 inobtcount=0
2024-05-08T01:13:22.197 INFO:teuthology.orchestra.run.smithi106.stdout:data     =                       bsize=4096   blocks=23436288, imaxpct=25
2024-05-08T01:13:22.197 INFO:teuthology.orchestra.run.smithi106.stdout:         =                       sunit=0      swidth=0 blks
2024-05-08T01:13:22.198 INFO:teuthology.orchestra.run.smithi106.stdout:naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
2024-05-08T01:13:22.198 INFO:teuthology.orchestra.run.smithi106.stdout:log      =internal log           bsize=4096   blocks=11443, version=2
2024-05-08T01:13:22.198 INFO:teuthology.orchestra.run.smithi106.stdout:         =                       sectsz=512   sunit=0 blks, lazy-count=1
2024-05-08T01:13:22.198 INFO:teuthology.orchestra.run.smithi106.stdout:realtime =none                   extsz=4096   blocks=0, rtextents=0
2024-05-08T01:13:22.668 INFO:teuthology.orchestra.run.smithi106.stdout:Discarding blocks...Done.
2024-05-08T01:13:22.670 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi106.front.sepia.ceph.com -o noatime
2024-05-08T01:13:22.670 DEBUG:teuthology.orchestra.run.smithi106:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2
2024-05-08T01:13:22.824 DEBUG:teuthology.orchestra.run.smithi106:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2
2024-05-08T01:13:22.835 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: /sbin/restorecon: command not found
2024-05-08T01:13:22.835 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T01:13:22.836 DEBUG:teuthology.orchestra.run.smithi106:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap
2024-05-08T01:13:22.957 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-08T01:13:22.973+0000 7fc4ed86f7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory
2024-05-08T01:13:22.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-08T01:13:22.977+0000 7fc4ed86f7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring
2024-05-08T01:13:22.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-08T01:13:22.977+0000 7fc4ed86f7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory
2024-05-08T01:13:22.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-08T01:13:22.977+0000 7fc4ed86f7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory
2024-05-08T01:13:22.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-08T01:13:22.977+0000 7fc4ed86f7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory
2024-05-08T01:13:22.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-08T01:13:22.977+0000 7fc4ed86f7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid
2024-05-08T01:13:25.931 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0
2024-05-08T01:13:25.945 DEBUG:teuthology.orchestra.run.smithi106:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap
2024-05-08T01:13:26.027 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-08T01:13:26.045+0000 7fc8641587c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory
2024-05-08T01:13:26.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-08T01:13:26.045+0000 7fc8641587c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring
2024-05-08T01:13:26.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-08T01:13:26.045+0000 7fc8641587c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory
2024-05-08T01:13:26.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-08T01:13:26.045+0000 7fc8641587c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory
2024-05-08T01:13:26.028 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-08T01:13:26.045+0000 7fc8641587c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory
2024-05-08T01:13:26.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-08T01:13:26.045+0000 7fc8641587c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid
2024-05-08T01:13:29.019 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1
2024-05-08T01:13:29.075 DEBUG:teuthology.orchestra.run.smithi106:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap
2024-05-08T01:13:29.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-08T01:13:29.173+0000 7feaf6c257c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory
2024-05-08T01:13:29.156 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-08T01:13:29.173+0000 7feaf6c257c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring
2024-05-08T01:13:29.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-08T01:13:29.173+0000 7feaf6c257c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory
2024-05-08T01:13:29.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-08T01:13:29.173+0000 7feaf6c257c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory
2024-05-08T01:13:29.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-08T01:13:29.173+0000 7feaf6c257c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory
2024-05-08T01:13:29.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-08T01:13:29.173+0000 7feaf6c257c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid
2024-05-08T01:13:32.136 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2
2024-05-08T01:13:32.191 INFO:tasks.ceph:Reading keys from all nodes...
2024-05-08T01:13:32.192 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:32.192 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout
2024-05-08T01:13:32.245 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:32.245 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout
2024-05-08T01:13:32.303 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:32.303 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout
2024-05-08T01:13:32.360 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:32.361 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout
2024-05-08T01:13:32.413 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:32.413 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout
2024-05-08T01:13:32.463 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:32.463 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout
2024-05-08T01:13:32.469 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:32.469 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout
2024-05-08T01:13:32.515 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:32.516 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout
2024-05-08T01:13:32.563 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:32.563 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout
2024-05-08T01:13:32.611 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:32.611 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout
2024-05-08T01:13:32.659 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:32.659 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout
2024-05-08T01:13:32.707 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:32.707 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout
2024-05-08T01:13:32.755 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:32.756 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout
2024-05-08T01:13:32.804 INFO:tasks.ceph:Adding keys to all mons...
2024-05-08T01:13:32.804 DEBUG:teuthology.orchestra.run.smithi106:> sudo tee -a /etc/ceph/cluster1.keyring
2024-05-08T01:13:32.816 INFO:teuthology.orchestra.run.smithi106.stdout:[mgr.x]
2024-05-08T01:13:32.816 INFO:teuthology.orchestra.run.smithi106.stdout:	key = AQCv0TpmyChwOxAAx7aVeQQM4p/RSJMAsneF7w==
2024-05-08T01:13:32.816 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.0]
2024-05-08T01:13:32.816 INFO:teuthology.orchestra.run.smithi106.stdout:	key = AQCy0Tpmr9taOhAAaSkOMuygKvVBjx8FM4wYfw==
2024-05-08T01:13:32.816 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.1]
2024-05-08T01:13:32.816 INFO:teuthology.orchestra.run.smithi106.stdout:	key = AQC20TpmFqvyAhAAaOX0NEhF3/zDAxOr2mkZfA==
2024-05-08T01:13:32.816 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.2]
2024-05-08T01:13:32.816 INFO:teuthology.orchestra.run.smithi106.stdout:	key = AQC50TpmhdCdChAAlJYKkdgNzRKcETUEeA9ZBA==
2024-05-08T01:13:32.816 INFO:teuthology.orchestra.run.smithi106.stdout:[client.0]
2024-05-08T01:13:32.817 INFO:teuthology.orchestra.run.smithi106.stdout:	key = AQCw0Tpm8YGDBBAAvGs9qiqo9INE096cQwHndg==
2024-05-08T01:13:32.817 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror]
2024-05-08T01:13:32.817 INFO:teuthology.orchestra.run.smithi106.stdout:	key = AQCw0TpmuEY9BxAA8mF/n1ILdufosemF9zXDqQ==
2024-05-08T01:13:32.817 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.0]
2024-05-08T01:13:32.817 INFO:teuthology.orchestra.run.smithi106.stdout:	key = AQCw0TpmKDRoDRAAkLFmFOTGyffdUBRbQo1QZQ==
2024-05-08T01:13:32.817 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.1]
2024-05-08T01:13:32.817 INFO:teuthology.orchestra.run.smithi106.stdout:	key = AQCw0TpmpBpHEhAAjBfzt3Kt3/GtZFeRBPopvA==
2024-05-08T01:13:32.817 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.2]
2024-05-08T01:13:32.817 INFO:teuthology.orchestra.run.smithi106.stdout:	key = AQCw0TpmRgxKFxAA6BY1rU8lpefCMcZbfYMIUw==
2024-05-08T01:13:32.817 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.3]
2024-05-08T01:13:32.817 INFO:teuthology.orchestra.run.smithi106.stdout:	key = AQCw0TpmpOVZHBAAhgCKU6ZG61kmmQ+H9ubnLQ==
2024-05-08T01:13:32.817 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.4]
2024-05-08T01:13:32.817 INFO:teuthology.orchestra.run.smithi106.stdout:	key = AQCw0TpmqGRXIRAAlU/iQY4uUexE/s/6Np/5+w==
2024-05-08T01:13:32.817 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.5]
2024-05-08T01:13:32.818 INFO:teuthology.orchestra.run.smithi106.stdout:	key = AQCw0Tpmj0c/JhAAhBhBGcmNBHfPQbMZ+etjwg==
2024-05-08T01:13:32.818 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.6]
2024-05-08T01:13:32.818 INFO:teuthology.orchestra.run.smithi106.stdout:	key = AQCw0TpmGdLqKhAAwWDHt75drquNIqZdMDBOHg==
2024-05-08T01:13:32.819 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
2024-05-08T01:13:32.906 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *'
2024-05-08T01:13:32.949 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *'
2024-05-08T01:13:33.029 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *'
2024-05-08T01:13:33.114 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow
2024-05-08T01:13:33.203 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow
2024-05-08T01:13:33.245 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow
2024-05-08T01:13:33.329 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow
2024-05-08T01:13:33.410 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow
2024-05-08T01:13:33.494 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow
2024-05-08T01:13:33.582 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow
2024-05-08T01:13:33.625 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow
2024-05-08T01:13:33.710 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow
2024-05-08T01:13:33.797 INFO:tasks.ceph:Running mkfs on mon nodes...
2024-05-08T01:13:33.797 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a
2024-05-08T01:13:33.854 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring
2024-05-08T01:13:33.946 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a
2024-05-08T01:13:33.957 DEBUG:teuthology.orchestra.run.smithi106:> rm -- /home/ubuntu/cephtest/cluster1.monmap
2024-05-08T01:13:34.001 INFO:tasks.ceph:Starting mon daemons in cluster cluster1...
2024-05-08T01:13:34.002 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon
2024-05-08T01:13:34.002 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a
2024-05-08T01:13:34.047 INFO:tasks.ceph.cluster1.mon.a:Started
2024-05-08T01:13:34.047 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1...
2024-05-08T01:13:34.047 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon
2024-05-08T01:13:34.047 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x
2024-05-08T01:13:34.050 INFO:tasks.ceph.cluster1.mgr.x:Started
2024-05-08T01:13:34.050 INFO:tasks.ceph:Setting crush tunables to default
2024-05-08T01:13:34.050 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster cluster1 osd crush tunables default
2024-05-08T01:13:34.276 INFO:teuthology.orchestra.run.smithi106.stderr:adjusted tunables profile to default
2024-05-08T01:13:34.294 INFO:tasks.ceph:check_enable_crimson: False
2024-05-08T01:13:34.294 INFO:tasks.ceph:Starting osd daemons in cluster cluster1...
2024-05-08T01:13:34.295 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:34.295 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout
2024-05-08T01:13:34.305 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:34.305 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout
2024-05-08T01:13:34.359 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:34.359 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout
2024-05-08T01:13:34.417 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster cluster1 osd new 3379a737-51ca-4783-a803-08467868290b 0
2024-05-08T01:13:34.431 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:34.449+0000 7eff50162180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member
2024-05-08T01:13:34.627 INFO:teuthology.orchestra.run.smithi106.stdout:0
2024-05-08T01:13:34.644 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster cluster1 osd new 750e4933-f084-4144-96fc-82a2cd313ffc 1
2024-05-08T01:13:34.838 INFO:teuthology.orchestra.run.smithi106.stdout:1
2024-05-08T01:13:34.855 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster cluster1 osd new 8b237cc9-901b-4426-9a3f-a8c1b215f2a3 2
2024-05-08T01:13:34.936 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:34.953+0000 7eff50162180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member
2024-05-08T01:13:35.040 INFO:teuthology.orchestra.run.smithi106.stdout:2
2024-05-08T01:13:35.056 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon
2024-05-08T01:13:35.057 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0
2024-05-08T01:13:35.059 INFO:tasks.ceph.cluster1.osd.0:Started
2024-05-08T01:13:35.059 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon
2024-05-08T01:13:35.059 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1
2024-05-08T01:13:35.060 INFO:tasks.ceph.cluster1.osd.1:Started
2024-05-08T01:13:35.061 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon
2024-05-08T01:13:35.061 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2
2024-05-08T01:13:35.062 INFO:tasks.ceph.cluster1.osd.2:Started
2024-05-08T01:13:35.063 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json
2024-05-08T01:13:35.139 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:35.157+0000 7eff50162180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member
2024-05-08T01:13:35.244 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:35.244 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":5,"fsid":"3284bad2-5fe9-40c3-bba2-9a50da63f434","created":"2024-05-08T01:13:34.190730+0000","modified":"2024-05-08T01:13:35.061884+0000","last_up_change":"0.000000","last_in_change":"2024-05-08T01:13:35.061884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3379a737-51ca-4783-a803-08467868290b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"750e4933-f084-4144-96fc-82a2cd313ffc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8b237cc9-901b-4426-9a3f-a8c1b215f2a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
2024-05-08T01:13:35.259 INFO:tasks.ceph.ceph_manager.cluster1:[]
2024-05-08T01:13:35.260 INFO:tasks.ceph:Waiting for OSDs to come up
2024-05-08T01:13:35.563 DEBUG:teuthology.orchestra.run.smithi106:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json
2024-05-08T01:13:35.686 INFO:tasks.ceph.cluster1.osd.1.smithi106.stderr:2024-05-08T01:13:35.704+0000 7f14435337c0 -1 Falling back to public interface
2024-05-08T01:13:35.690 INFO:tasks.ceph.cluster1.osd.2.smithi106.stderr:2024-05-08T01:13:35.708+0000 7fe4e4f067c0 -1 Falling back to public interface
2024-05-08T01:13:35.697 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:35.712+0000 7eff50162180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member
2024-05-08T01:13:35.702 INFO:tasks.ceph.cluster1.osd.0.smithi106.stderr:2024-05-08T01:13:35.720+0000 7f9af6bee7c0 -1 Falling back to public interface
2024-05-08T01:13:35.744 INFO:teuthology.misc.health.smithi106.stdout:
2024-05-08T01:13:35.744 INFO:teuthology.misc.health.smithi106.stdout:{"epoch":5,"fsid":"3284bad2-5fe9-40c3-bba2-9a50da63f434","created":"2024-05-08T01:13:34.190730+0000","modified":"2024-05-08T01:13:35.061884+0000","last_up_change":"0.000000","last_in_change":"2024-05-08T01:13:35.061884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3379a737-51ca-4783-a803-08467868290b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"750e4933-f084-4144-96fc-82a2cd313ffc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8b237cc9-901b-4426-9a3f-a8c1b215f2a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
2024-05-08T01:13:35.753 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:35.768+0000 7eff50162180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member
2024-05-08T01:13:35.759 DEBUG:teuthology.misc:0 of 3 OSDs are up
2024-05-08T01:13:35.806 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:35.824+0000 7eff50162180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member
2024-05-08T01:13:35.882 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:35.900+0000 7eff50162180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member
2024-05-08T01:13:35.933 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:35.948+0000 7eff50162180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member
2024-05-08T01:13:36.009 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:36.028+0000 7eff50162180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member
2024-05-08T01:13:36.062 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:36.080+0000 7eff50162180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member
2024-05-08T01:13:36.211 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:36.228+0000 7eff50162180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member
2024-05-08T01:13:36.271 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:36.288+0000 7eff50162180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member
2024-05-08T01:13:36.866 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode.
2024-05-08T01:13:36.866 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve.
2024-05-08T01:13:36.867 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:  from numpy import show_config as show_numpy_config
2024-05-08T01:13:36.932 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:36.948+0000 7eff50162180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member
2024-05-08T01:13:37.108 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:37.124+0000 7eff50162180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member
2024-05-08T01:13:37.265 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:37.284+0000 7eff50162180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member
2024-05-08T01:13:37.336 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:37.352+0000 7eff50162180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member
2024-05-08T01:13:37.472 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:37.488+0000 7eff50162180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member
2024-05-08T01:13:38.087 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:38.104+0000 7eff50162180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member
2024-05-08T01:13:38.162 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:38.180+0000 7eff50162180 -1 mgr[py] Module status has missing NOTIFY_TYPES member
2024-05-08T01:13:38.341 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:38.360+0000 7eff50162180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member
2024-05-08T01:13:38.416 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:38.432+0000 7eff50162180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member
2024-05-08T01:13:38.708 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:38.724+0000 7eff50162180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member
2024-05-08T01:13:38.773 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:38.792+0000 7eff50162180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member
2024-05-08T01:13:39.509 INFO:tasks.ceph.cluster1.osd.2.smithi106.stderr:2024-05-08T01:13:39.524+0000 7fe4e4f067c0 -1 osd.2 0 log_to_monitors true
2024-05-08T01:13:39.599 INFO:tasks.ceph.cluster1.osd.1.smithi106.stderr:2024-05-08T01:13:39.616+0000 7f14435337c0 -1 osd.1 0 log_to_monitors true
2024-05-08T01:13:39.599 INFO:tasks.ceph.cluster1.osd.0.smithi106.stderr:2024-05-08T01:13:39.616+0000 7f9af6bee7c0 -1 osd.0 0 log_to_monitors true
2024-05-08T01:13:39.809 INFO:tasks.ceph.cluster1.mgr.x.smithi106.stderr:2024-05-08T01:13:39.824+0000 7efef75a9640 -1 mgr.server handle_report got status from non-daemon mon.a
2024-05-08T01:13:42.063 DEBUG:teuthology.orchestra.run.smithi106:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json
2024-05-08T01:13:42.488 INFO:teuthology.misc.health.smithi106.stdout:
2024-05-08T01:13:42.489 INFO:teuthology.misc.health.smithi106.stdout:{"epoch":7,"fsid":"3284bad2-5fe9-40c3-bba2-9a50da63f434","created":"2024-05-08T01:13:34.190730+0000","modified":"2024-05-08T01:13:40.880106+0000","last_up_change":"0.000000","last_in_change":"2024-05-08T01:13:35.061884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3379a737-51ca-4783-a803-08467868290b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"750e4933-f084-4144-96fc-82a2cd313ffc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8b237cc9-901b-4426-9a3f-a8c1b215f2a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
2024-05-08T01:13:42.512 DEBUG:teuthology.misc:0 of 3 OSDs are up
2024-05-08T01:13:42.758 INFO:tasks.ceph.cluster1.osd.2.smithi106.stderr:2024-05-08T01:13:42.776+0000 7fe4e16c1640 -1 osd.2 0 waiting for initial osdmap
2024-05-08T01:13:42.782 INFO:tasks.ceph.cluster1.osd.2.smithi106.stderr:2024-05-08T01:13:42.800+0000 7fe4d8ce4640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory
2024-05-08T01:13:42.799 INFO:tasks.ceph.cluster1.osd.1.smithi106.stderr:2024-05-08T01:13:42.816+0000 7f143f4dc640 -1 osd.1 0 waiting for initial osdmap
2024-05-08T01:13:42.801 INFO:tasks.ceph.cluster1.osd.0.smithi106.stderr:2024-05-08T01:13:42.816+0000 7f9af33a9640 -1 osd.0 0 waiting for initial osdmap
2024-05-08T01:13:42.826 INFO:tasks.ceph.cluster1.osd.1.smithi106.stderr:2024-05-08T01:13:42.844+0000 7f1437311640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory
2024-05-08T01:13:42.838 INFO:tasks.ceph.cluster1.osd.0.smithi106.stderr:2024-05-08T01:13:42.856+0000 7f9aea9cc640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory
2024-05-08T01:13:48.815 DEBUG:teuthology.orchestra.run.smithi106:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json
2024-05-08T01:13:49.115 INFO:teuthology.misc.health.smithi106.stdout:
2024-05-08T01:13:49.116 INFO:teuthology.misc.health.smithi106.stdout:{"epoch":11,"fsid":"3284bad2-5fe9-40c3-bba2-9a50da63f434","created":"2024-05-08T01:13:34.190730+0000","modified":"2024-05-08T01:13:46.913197+0000","last_up_change":"2024-05-08T01:13:43.826579+0000","last_in_change":"2024-05-08T01:13:35.061884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T01:13:44.899677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3379a737-51ca-4783-a803-08467868290b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3953308227},{"type":"v1","addr":"172.21.15.106:6817","nonce":3953308227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":3953308227},{"type":"v1","addr":"172.21.15.106:6819","nonce":3953308227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":3953308227},{"type":"v1","addr":"172.21.15.106:6823","nonce":3953308227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":3953308227},{"type":"v1","addr":"172.21.15.106:6821","nonce":3953308227}]},"public_addr":"172.21.15.106:6817/3953308227","cluster_addr":"172.21.15.106:6819/3953308227","heartbeat_back_addr":"172.21.15.106:6823/3953308227","heartbeat_front_addr":"172.21.15.106:6821/3953308227","state":["exists","up"]},{"osd":1,"uuid":"750e4933-f084-4144-96fc-82a2cd313ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":4128478887},{"type":"v1","addr":"172.21.15.106:6801","nonce":4128478887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":4128478887},{"type":"v1","addr":"172.21.15.106:6803","nonce":4128478887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":4128478887},{"type":"v1","addr":"172.21.15.106:6807","nonce":4128478887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":4128478887},{"type":"v1","addr":"172.21.15.106:6805","nonce":4128478887}]},"public_addr":"172.21.15.106:6801/4128478887","cluster_addr":"172.21.15.106:6803/4128478887","heartbeat_back_addr":"172.21.15.106:6807/4128478887","heartbeat_front_addr":"172.21.15.106:6805/4128478887","state":["exists","up"]},{"osd":2,"uuid":"8b237cc9-901b-4426-9a3f-a8c1b215f2a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1358302753},{"type":"v1","addr":"172.21.15.106:6809","nonce":1358302753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":1358302753},{"type":"v1","addr":"172.21.15.106:6811","nonce":1358302753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":1358302753},{"type":"v1","addr":"172.21.15.106:6815","nonce":1358302753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":1358302753},{"type":"v1","addr":"172.21.15.106:6813","nonce":1358302753}]},"public_addr":"172.21.15.106:6809/1358302753","cluster_addr":"172.21.15.106:6811/1358302753","heartbeat_back_addr":"172.21.15.106:6815/1358302753","heartbeat_front_addr":"172.21.15.106:6813/1358302753","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:13:40.615948+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:13:40.595389+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:13:40.580330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
2024-05-08T01:13:49.132 DEBUG:teuthology.misc:3 of 3 OSDs are up
2024-05-08T01:13:49.133 INFO:tasks.ceph:Creating RBD pool
2024-05-08T01:13:49.133 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster cluster1 osd pool create rbd 8
2024-05-08T01:13:49.907 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'rbd' created
2024-05-08T01:13:49.932 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it
2024-05-08T01:13:50.908 INFO:teuthology.orchestra.run.smithi106.stderr:enabled application 'rbd' on pool 'rbd'
2024-05-08T01:13:50.936 INFO:tasks.ceph:Starting mds daemons in cluster cluster1...
2024-05-08T01:13:50.936 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean...
2024-05-08T01:13:50.937 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available
2024-05-08T01:13:50.937 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json
2024-05-08T01:13:50.937 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting
2024-05-08T01:13:51.266 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:51.282 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":2396069218},{"type":"v1","addr":"172.21.15.106:6825","nonce":2396069218}]},"active_addr":"172.21.15.106:6825/2396069218","active_change":"2024-05-08T01:13:38.817017+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace.  Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB.  Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":699468801}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":2343964769}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":4079122895}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":782987067}]}]}
2024-05-08T01:13:51.285 INFO:tasks.ceph.ceph_manager.cluster1:mgr available!
2024-05-08T01:13:51.285 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up
2024-05-08T01:13:51.286 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json
2024-05-08T01:13:51.568 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:51.568 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":13,"fsid":"3284bad2-5fe9-40c3-bba2-9a50da63f434","created":"2024-05-08T01:13:34.190730+0000","modified":"2024-05-08T01:13:50.929069+0000","last_up_change":"2024-05-08T01:13:43.826579+0000","last_in_change":"2024-05-08T01:13:35.061884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T01:13:44.899677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-08T01:13:49.489312+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3379a737-51ca-4783-a803-08467868290b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3953308227},{"type":"v1","addr":"172.21.15.106:6817","nonce":3953308227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":3953308227},{"type":"v1","addr":"172.21.15.106:6819","nonce":3953308227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":3953308227},{"type":"v1","addr":"172.21.15.106:6823","nonce":3953308227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":3953308227},{"type":"v1","addr":"172.21.15.106:6821","nonce":3953308227}]},"public_addr":"172.21.15.106:6817/3953308227","cluster_addr":"172.21.15.106:6819/3953308227","heartbeat_back_addr":"172.21.15.106:6823/3953308227","heartbeat_front_addr":"172.21.15.106:6821/3953308227","state":["exists","up"]},{"osd":1,"uuid":"750e4933-f084-4144-96fc-82a2cd313ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":4128478887},{"type":"v1","addr":"172.21.15.106:6801","nonce":4128478887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":4128478887},{"type":"v1","addr":"172.21.15.106:6803","nonce":4128478887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":4128478887},{"type":"v1","addr":"172.21.15.106:6807","nonce":4128478887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":4128478887},{"type":"v1","addr":"172.21.15.106:6805","nonce":4128478887}]},"public_addr":"172.21.15.106:6801/4128478887","cluster_addr":"172.21.15.106:6803/4128478887","heartbeat_back_addr":"172.21.15.106:6807/4128478887","heartbeat_front_addr":"172.21.15.106:6805/4128478887","state":["exists","up"]},{"osd":2,"uuid":"8b237cc9-901b-4426-9a3f-a8c1b215f2a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1358302753},{"type":"v1","addr":"172.21.15.106:6809","nonce":1358302753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":1358302753},{"type":"v1","addr":"172.21.15.106:6811","nonce":1358302753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":1358302753},{"type":"v1","addr":"172.21.15.106:6815","nonce":1358302753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":1358302753},{"type":"v1","addr":"172.21.15.106:6813","nonce":1358302753}]},"public_addr":"172.21.15.106:6809/1358302753","cluster_addr":"172.21.15.106:6811/1358302753","heartbeat_back_addr":"172.21.15.106:6815/1358302753","heartbeat_front_addr":"172.21.15.106:6813/1358302753","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:13:40.615948+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:13:40.595389+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:13:40.580330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
2024-05-08T01:13:51.586 INFO:tasks.ceph.ceph_manager.cluster1:all up!
2024-05-08T01:13:51.586 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json
2024-05-08T01:13:51.857 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:51.857 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":13,"fsid":"3284bad2-5fe9-40c3-bba2-9a50da63f434","created":"2024-05-08T01:13:34.190730+0000","modified":"2024-05-08T01:13:50.929069+0000","last_up_change":"2024-05-08T01:13:43.826579+0000","last_in_change":"2024-05-08T01:13:35.061884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T01:13:44.899677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-08T01:13:49.489312+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3379a737-51ca-4783-a803-08467868290b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3953308227},{"type":"v1","addr":"172.21.15.106:6817","nonce":3953308227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":3953308227},{"type":"v1","addr":"172.21.15.106:6819","nonce":3953308227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":3953308227},{"type":"v1","addr":"172.21.15.106:6823","nonce":3953308227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":3953308227},{"type":"v1","addr":"172.21.15.106:6821","nonce":3953308227}]},"public_addr":"172.21.15.106:6817/3953308227","cluster_addr":"172.21.15.106:6819/3953308227","heartbeat_back_addr":"172.21.15.106:6823/3953308227","heartbeat_front_addr":"172.21.15.106:6821/3953308227","state":["exists","up"]},{"osd":1,"uuid":"750e4933-f084-4144-96fc-82a2cd313ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":4128478887},{"type":"v1","addr":"172.21.15.106:6801","nonce":4128478887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":4128478887},{"type":"v1","addr":"172.21.15.106:6803","nonce":4128478887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":4128478887},{"type":"v1","addr":"172.21.15.106:6807","nonce":4128478887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":4128478887},{"type":"v1","addr":"172.21.15.106:6805","nonce":4128478887}]},"public_addr":"172.21.15.106:6801/4128478887","cluster_addr":"172.21.15.106:6803/4128478887","heartbeat_back_addr":"172.21.15.106:6807/4128478887","heartbeat_front_addr":"172.21.15.106:6805/4128478887","state":["exists","up"]},{"osd":2,"uuid":"8b237cc9-901b-4426-9a3f-a8c1b215f2a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1358302753},{"type":"v1","addr":"172.21.15.106:6809","nonce":1358302753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":1358302753},{"type":"v1","addr":"172.21.15.106:6811","nonce":1358302753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":1358302753},{"type":"v1","addr":"172.21.15.106:6815","nonce":1358302753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":1358302753},{"type":"v1","addr":"172.21.15.106:6813","nonce":1358302753}]},"public_addr":"172.21.15.106:6809/1358302753","cluster_addr":"172.21.15.106:6811/1358302753","heartbeat_back_addr":"172.21.15.106:6815/1358302753","heartbeat_front_addr":"172.21.15.106:6813/1358302753","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:13:40.615948+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:13:40.595389+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:13:40.580330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
2024-05-08T01:13:51.874 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats
2024-05-08T01:13:51.875 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats
2024-05-08T01:13:51.875 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats
2024-05-08T01:13:52.021 INFO:teuthology.orchestra.run.smithi106.stdout:34359738372
2024-05-08T01:13:52.021 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0
2024-05-08T01:13:52.037 INFO:teuthology.orchestra.run.smithi106.stdout:34359738373
2024-05-08T01:13:52.038 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1
2024-05-08T01:13:52.040 INFO:teuthology.orchestra.run.smithi106.stdout:34359738372
2024-05-08T01:13:52.040 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2
2024-05-08T01:13:52.344 INFO:teuthology.orchestra.run.smithi106.stdout:34359738371
2024-05-08T01:13:52.354 INFO:teuthology.orchestra.run.smithi106.stdout:34359738372
2024-05-08T01:13:52.362 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.0
2024-05-08T01:13:52.372 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.1
2024-05-08T01:13:52.400 INFO:teuthology.orchestra.run.smithi106.stdout:34359738371
2024-05-08T01:13:52.416 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.2
2024-05-08T01:13:53.363 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0
2024-05-08T01:13:53.372 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1
2024-05-08T01:13:53.417 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2
2024-05-08T01:13:53.678 INFO:teuthology.orchestra.run.smithi106.stdout:34359738372
2024-05-08T01:13:53.695 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.0
2024-05-08T01:13:53.695 DEBUG:teuthology.parallel:result is None
2024-05-08T01:13:53.697 INFO:teuthology.orchestra.run.smithi106.stdout:34359738373
2024-05-08T01:13:53.714 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.1
2024-05-08T01:13:53.714 DEBUG:teuthology.parallel:result is None
2024-05-08T01:13:53.736 INFO:teuthology.orchestra.run.smithi106.stdout:34359738372
2024-05-08T01:13:53.752 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.2
2024-05-08T01:13:53.752 DEBUG:teuthology.parallel:result is None
2024-05-08T01:13:53.752 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean
2024-05-08T01:13:53.752 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json
2024-05-08T01:13:54.085 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:54.086 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all
2024-05-08T01:13:54.104 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-05-08T01:13:52.826937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82564,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032956,"statfs":{"total":289910292480,"available":289825746944,"internally_reserved":0,"allocated":1531904,"data_stored":1274801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.896830"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T01:13:50.947542+0000","last_change":"2024-05-08T01:13:50.943750+0000","last_active":"2024-05-08T01:13:50.947542+0000","last_peered":"2024-05-08T01:13:50.947542+0000","last_clean":"2024-05-08T01:13:50.947542+0000","last_became_active":"2024-05-08T01:13:50.943014+0000","last_became_peered":"2024-05-08T01:13:50.943014+0000","last_unstale":"2024-05-08T01:13:50.947542+0000","last_undegraded":"2024-05-08T01:13:50.947542+0000","last_fullsized":"2024-05-08T01:13:50.947542+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T11:41:34.770642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T01:13:50.949974+0000","last_change":"2024-05-08T01:13:50.945953+0000","last_active":"2024-05-08T01:13:50.949974+0000","last_peered":"2024-05-08T01:13:50.949974+0000","last_clean":"2024-05-08T01:13:50.949974+0000","last_became_active":"2024-05-08T01:13:50.945427+0000","last_became_peered":"2024-05-08T01:13:50.945427+0000","last_unstale":"2024-05-08T01:13:50.949974+0000","last_undegraded":"2024-05-08T01:13:50.949974+0000","last_fullsized":"2024-05-08T01:13:50.949974+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:48:04.721848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T01:13:50.947585+0000","last_change":"2024-05-08T01:13:50.943381+0000","last_active":"2024-05-08T01:13:50.947585+0000","last_peered":"2024-05-08T01:13:50.947585+0000","last_clean":"2024-05-08T01:13:50.947585+0000","last_became_active":"2024-05-08T01:13:50.942923+0000","last_became_peered":"2024-05-08T01:13:50.942923+0000","last_unstale":"2024-05-08T01:13:50.947585+0000","last_undegraded":"2024-05-08T01:13:50.947585+0000","last_fullsized":"2024-05-08T01:13:50.947585+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T11:23:52.237680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T01:13:50.950050+0000","last_change":"2024-05-08T01:13:50.946454+0000","last_active":"2024-05-08T01:13:50.950050+0000","last_peered":"2024-05-08T01:13:50.950050+0000","last_clean":"2024-05-08T01:13:50.950050+0000","last_became_active":"2024-05-08T01:13:50.945779+0000","last_became_peered":"2024-05-08T01:13:50.945779+0000","last_unstale":"2024-05-08T01:13:50.950050+0000","last_undegraded":"2024-05-08T01:13:50.950050+0000","last_fullsized":"2024-05-08T01:13:50.950050+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:05:47.504933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T01:13:51.938834+0000","last_change":"2024-05-08T01:13:50.945406+0000","last_active":"2024-05-08T01:13:51.938834+0000","last_peered":"2024-05-08T01:13:51.938834+0000","last_clean":"2024-05-08T01:13:51.938834+0000","last_became_active":"2024-05-08T01:13:50.944956+0000","last_became_peered":"2024-05-08T01:13:50.944956+0000","last_unstale":"2024-05-08T01:13:51.938834+0000","last_undegraded":"2024-05-08T01:13:51.938834+0000","last_fullsized":"2024-05-08T01:13:51.938834+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:06:46.842113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T01:13:50.947542+0000","last_change":"2024-05-08T01:13:50.946179+0000","last_active":"2024-05-08T01:13:50.947542+0000","last_peered":"2024-05-08T01:13:50.947542+0000","last_clean":"2024-05-08T01:13:50.947542+0000","last_became_active":"2024-05-08T01:13:50.945466+0000","last_became_peered":"2024-05-08T01:13:50.945466+0000","last_unstale":"2024-05-08T01:13:50.947542+0000","last_undegraded":"2024-05-08T01:13:50.947542+0000","last_fullsized":"2024-05-08T01:13:50.947542+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:05:11.335580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T01:13:50.947511+0000","last_change":"2024-05-08T01:13:50.945971+0000","last_active":"2024-05-08T01:13:50.947511+0000","last_peered":"2024-05-08T01:13:50.947511+0000","last_clean":"2024-05-08T01:13:50.947511+0000","last_became_active":"2024-05-08T01:13:50.945591+0000","last_became_peered":"2024-05-08T01:13:50.945591+0000","last_unstale":"2024-05-08T01:13:50.947511+0000","last_undegraded":"2024-05-08T01:13:50.947511+0000","last_fullsized":"2024-05-08T01:13:50.947511+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:15:16.944698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T01:13:50.947488+0000","last_change":"2024-05-08T01:13:50.942235+0000","last_active":"2024-05-08T01:13:50.947488+0000","last_peered":"2024-05-08T01:13:50.947488+0000","last_clean":"2024-05-08T01:13:50.947488+0000","last_became_active":"2024-05-08T01:13:50.941816+0000","last_became_peered":"2024-05-08T01:13:50.941816+0000","last_unstale":"2024-05-08T01:13:50.947488+0000","last_undegraded":"2024-05-08T01:13:50.947488+0000","last_fullsized":"2024-05-08T01:13:50.947488+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:31:31.514514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":143,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T01:13:50.933970+0000","last_change":"2024-05-08T01:13:45.924420+0000","last_active":"2024-05-08T01:13:50.933970+0000","last_peered":"2024-05-08T01:13:50.933970+0000","last_clean":"2024-05-08T01:13:50.933970+0000","last_became_active":"2024-05-08T01:13:45.921946+0000","last_became_peered":"2024-05-08T01:13:45.921946+0000","last_unstale":"2024-05-08T01:13:50.933970+0000","last_undegraded":"2024-05-08T01:13:50.933970+0000","last_fullsized":"2024-05-08T01:13:50.933970+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:44.905674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:44.905674+0000","last_clean_scrub_stamp":"2024-05-08T01:13:44.905674+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:33:37.631809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}}
2024-05-08T01:13:54.105 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json
2024-05-08T01:13:54.380 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:54.380 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all
2024-05-08T01:13:54.396 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-05-08T01:13:52.826937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82564,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032956,"statfs":{"total":289910292480,"available":289825746944,"internally_reserved":0,"allocated":1531904,"data_stored":1274801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.896830"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T01:13:50.947542+0000","last_change":"2024-05-08T01:13:50.943750+0000","last_active":"2024-05-08T01:13:50.947542+0000","last_peered":"2024-05-08T01:13:50.947542+0000","last_clean":"2024-05-08T01:13:50.947542+0000","last_became_active":"2024-05-08T01:13:50.943014+0000","last_became_peered":"2024-05-08T01:13:50.943014+0000","last_unstale":"2024-05-08T01:13:50.947542+0000","last_undegraded":"2024-05-08T01:13:50.947542+0000","last_fullsized":"2024-05-08T01:13:50.947542+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T11:41:34.770642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T01:13:50.949974+0000","last_change":"2024-05-08T01:13:50.945953+0000","last_active":"2024-05-08T01:13:50.949974+0000","last_peered":"2024-05-08T01:13:50.949974+0000","last_clean":"2024-05-08T01:13:50.949974+0000","last_became_active":"2024-05-08T01:13:50.945427+0000","last_became_peered":"2024-05-08T01:13:50.945427+0000","last_unstale":"2024-05-08T01:13:50.949974+0000","last_undegraded":"2024-05-08T01:13:50.949974+0000","last_fullsized":"2024-05-08T01:13:50.949974+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:48:04.721848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T01:13:50.947585+0000","last_change":"2024-05-08T01:13:50.943381+0000","last_active":"2024-05-08T01:13:50.947585+0000","last_peered":"2024-05-08T01:13:50.947585+0000","last_clean":"2024-05-08T01:13:50.947585+0000","last_became_active":"2024-05-08T01:13:50.942923+0000","last_became_peered":"2024-05-08T01:13:50.942923+0000","last_unstale":"2024-05-08T01:13:50.947585+0000","last_undegraded":"2024-05-08T01:13:50.947585+0000","last_fullsized":"2024-05-08T01:13:50.947585+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T11:23:52.237680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T01:13:50.950050+0000","last_change":"2024-05-08T01:13:50.946454+0000","last_active":"2024-05-08T01:13:50.950050+0000","last_peered":"2024-05-08T01:13:50.950050+0000","last_clean":"2024-05-08T01:13:50.950050+0000","last_became_active":"2024-05-08T01:13:50.945779+0000","last_became_peered":"2024-05-08T01:13:50.945779+0000","last_unstale":"2024-05-08T01:13:50.950050+0000","last_undegraded":"2024-05-08T01:13:50.950050+0000","last_fullsized":"2024-05-08T01:13:50.950050+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:05:47.504933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T01:13:51.938834+0000","last_change":"2024-05-08T01:13:50.945406+0000","last_active":"2024-05-08T01:13:51.938834+0000","last_peered":"2024-05-08T01:13:51.938834+0000","last_clean":"2024-05-08T01:13:51.938834+0000","last_became_active":"2024-05-08T01:13:50.944956+0000","last_became_peered":"2024-05-08T01:13:50.944956+0000","last_unstale":"2024-05-08T01:13:51.938834+0000","last_undegraded":"2024-05-08T01:13:51.938834+0000","last_fullsized":"2024-05-08T01:13:51.938834+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:06:46.842113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T01:13:50.947542+0000","last_change":"2024-05-08T01:13:50.946179+0000","last_active":"2024-05-08T01:13:50.947542+0000","last_peered":"2024-05-08T01:13:50.947542+0000","last_clean":"2024-05-08T01:13:50.947542+0000","last_became_active":"2024-05-08T01:13:50.945466+0000","last_became_peered":"2024-05-08T01:13:50.945466+0000","last_unstale":"2024-05-08T01:13:50.947542+0000","last_undegraded":"2024-05-08T01:13:50.947542+0000","last_fullsized":"2024-05-08T01:13:50.947542+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:05:11.335580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T01:13:50.947511+0000","last_change":"2024-05-08T01:13:50.945971+0000","last_active":"2024-05-08T01:13:50.947511+0000","last_peered":"2024-05-08T01:13:50.947511+0000","last_clean":"2024-05-08T01:13:50.947511+0000","last_became_active":"2024-05-08T01:13:50.945591+0000","last_became_peered":"2024-05-08T01:13:50.945591+0000","last_unstale":"2024-05-08T01:13:50.947511+0000","last_undegraded":"2024-05-08T01:13:50.947511+0000","last_fullsized":"2024-05-08T01:13:50.947511+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:15:16.944698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T01:13:50.947488+0000","last_change":"2024-05-08T01:13:50.942235+0000","last_active":"2024-05-08T01:13:50.947488+0000","last_peered":"2024-05-08T01:13:50.947488+0000","last_clean":"2024-05-08T01:13:50.947488+0000","last_became_active":"2024-05-08T01:13:50.941816+0000","last_became_peered":"2024-05-08T01:13:50.941816+0000","last_unstale":"2024-05-08T01:13:50.947488+0000","last_undegraded":"2024-05-08T01:13:50.947488+0000","last_fullsized":"2024-05-08T01:13:50.947488+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:31:31.514514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":143,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T01:13:50.933970+0000","last_change":"2024-05-08T01:13:45.924420+0000","last_active":"2024-05-08T01:13:50.933970+0000","last_peered":"2024-05-08T01:13:50.933970+0000","last_clean":"2024-05-08T01:13:50.933970+0000","last_became_active":"2024-05-08T01:13:45.921946+0000","last_became_peered":"2024-05-08T01:13:45.921946+0000","last_unstale":"2024-05-08T01:13:50.933970+0000","last_undegraded":"2024-05-08T01:13:50.933970+0000","last_fullsized":"2024-05-08T01:13:50.933970+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:44.905674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:44.905674+0000","last_clean_scrub_stamp":"2024-05-08T01:13:44.905674+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:33:37.631809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}}
2024-05-08T01:13:54.397 INFO:tasks.ceph.ceph_manager.cluster1:clean!
2024-05-08T01:13:54.397 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy...
2024-05-08T01:13:54.397 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy
2024-05-08T01:13:54.397 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json
2024-05-08T01:13:54.706 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:54.706 INFO:teuthology.orchestra.run.smithi106.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]}
2024-05-08T01:13:54.723 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done
2024-05-08T01:13:54.724 INFO:teuthology.run_tasks:Running task ceph...
2024-05-08T01:13:54.734 INFO:tasks.ceph:Creating ceph cluster cluster2...
2024-05-08T01:13:54.734 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True}
2024-05-08T01:13:54.735 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696665', 'branch': 'wip-dis-testing-squid', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-policy-none}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7696665', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror.sh workunit to test the rbd-mirror daemon in journal mode'}], 'name': 'dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing-squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing-squid', 'sha1': '9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 5306, 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220', 'sleep_before_teardown': 0, 'subset': '21/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': '9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2', 'targets': {'smithi106.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEhs3TLZ8P9r3DOr+8IGGUHcnfAVbx7TqpskSVSFgCjPGwIhD09SIBtp+Z7rjLdnhz1N5Qf4WDMEJUB4B73G1wU=', 'smithi142.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAA1toN6R9IHXeOAxGgCT6AY7lk0P4OC4lEcl3mqalcJ3g8GXSNmbOrbouPGsGsi9R87RjctY7iw/lylaW12BKM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'journal', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '1ae7ad82388e92a475afff437d49054826c019a1', 'timestamp': '2024-05-07_23:02:08', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'}
2024-05-08T01:13:54.735 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data
2024-05-08T01:13:54.738 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data
2024-05-08T01:13:54.744 DEBUG:teuthology.orchestra.run.smithi106:> sudo install -d -m0777 -- /var/run/ceph
2024-05-08T01:13:54.786 DEBUG:teuthology.orchestra.run.smithi142:> sudo install -d -m0777 -- /var/run/ceph
2024-05-08T01:13:54.800 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:54.800 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/scratch_devs of=/dev/stdout
2024-05-08T01:13:54.847 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4']
2024-05-08T01:13:54.847 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_1
2024-05-08T01:13:54.895 INFO:teuthology.orchestra.run.smithi142.stdout:  File: /dev/vg_nvme/lv_1 -> ../dm-0
2024-05-08T01:13:54.895 INFO:teuthology.orchestra.run.smithi142.stdout:  Size: 7         	Blocks: 0          IO Block: 4096   symbolic link
2024-05-08T01:13:54.895 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d	Inode: 890         Links: 1
2024-05-08T01:13:54.895 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
2024-05-08T01:13:54.896 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-05-08 01:07:36.253781641 +0000
2024-05-08T01:13:54.896 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-05-08 01:07:35.969784178 +0000
2024-05-08T01:13:54.896 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-05-08 01:07:35.969784178 +0000
2024-05-08T01:13:54.896 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: -
2024-05-08T01:13:54.896 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1
2024-05-08T01:13:54.948 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in
2024-05-08T01:13:54.949 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out
2024-05-08T01:13:54.949 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000338488 s, 1.5 MB/s
2024-05-08T01:13:54.950 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1
2024-05-08T01:13:55.000 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_2
2024-05-08T01:13:55.047 INFO:teuthology.orchestra.run.smithi142.stdout:  File: /dev/vg_nvme/lv_2 -> ../dm-1
2024-05-08T01:13:55.047 INFO:teuthology.orchestra.run.smithi142.stdout:  Size: 7         	Blocks: 0          IO Block: 4096   symbolic link
2024-05-08T01:13:55.047 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d	Inode: 908         Links: 1
2024-05-08T01:13:55.047 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
2024-05-08T01:13:55.047 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-05-08 01:07:36.729777389 +0000
2024-05-08T01:13:55.047 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-05-08 01:07:36.489779533 +0000
2024-05-08T01:13:55.047 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-05-08 01:07:36.489779533 +0000
2024-05-08T01:13:55.047 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: -
2024-05-08T01:13:55.048 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1
2024-05-08T01:13:55.100 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in
2024-05-08T01:13:55.100 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out
2024-05-08T01:13:55.100 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000310222 s, 1.7 MB/s
2024-05-08T01:13:55.101 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2
2024-05-08T01:13:55.150 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_3
2024-05-08T01:13:55.199 INFO:teuthology.orchestra.run.smithi142.stdout:  File: /dev/vg_nvme/lv_3 -> ../dm-2
2024-05-08T01:13:55.199 INFO:teuthology.orchestra.run.smithi142.stdout:  Size: 7         	Blocks: 0          IO Block: 4096   symbolic link
2024-05-08T01:13:55.199 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d	Inode: 926         Links: 1
2024-05-08T01:13:55.199 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
2024-05-08T01:13:55.199 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-05-08 01:07:37.221772995 +0000
2024-05-08T01:13:55.199 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-05-08 01:07:36.965775281 +0000
2024-05-08T01:13:55.199 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-05-08 01:07:36.965775281 +0000
2024-05-08T01:13:55.199 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: -
2024-05-08T01:13:55.200 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1
2024-05-08T01:13:55.253 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in
2024-05-08T01:13:55.254 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out
2024-05-08T01:13:55.254 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000424017 s, 1.2 MB/s
2024-05-08T01:13:55.255 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3
2024-05-08T01:13:55.304 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_4
2024-05-08T01:13:55.351 INFO:teuthology.orchestra.run.smithi142.stdout:  File: /dev/vg_nvme/lv_4 -> ../dm-3
2024-05-08T01:13:55.351 INFO:teuthology.orchestra.run.smithi142.stdout:  Size: 7         	Blocks: 0          IO Block: 4096   symbolic link
2024-05-08T01:13:55.351 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d	Inode: 944         Links: 1
2024-05-08T01:13:55.351 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
2024-05-08T01:13:55.351 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-05-08 01:07:37.701768708 +0000
2024-05-08T01:13:55.351 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-05-08 01:07:37.445770994 +0000
2024-05-08T01:13:55.351 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-05-08 01:07:37.445770994 +0000
2024-05-08T01:13:55.352 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: -
2024-05-08T01:13:55.352 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1
2024-05-08T01:13:55.406 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in
2024-05-08T01:13:55.406 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out
2024-05-08T01:13:55.407 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000264283 s, 1.9 MB/s
2024-05-08T01:13:55.407 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4
2024-05-08T01:13:55.456 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}
2024-05-08T01:13:55.456 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi142.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}}
2024-05-08T01:13:55.456 INFO:tasks.ceph:Generating config...
2024-05-08T01:13:55.457 INFO:tasks.ceph:[client] debug rbd = 20
2024-05-08T01:13:55.457 INFO:tasks.ceph:[client] debug rbd_mirror = 15
2024-05-08T01:13:55.457 INFO:tasks.ceph:[client] log to stderr = False
2024-05-08T01:13:55.457 INFO:tasks.ceph:[client] rbd default features = 125
2024-05-08T01:13:55.457 INFO:tasks.ceph:[client] rbd mirror image policy type = none
2024-05-08T01:13:55.457 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.457 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.457 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.457 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.457 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.458 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.458 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.458 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.458 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.458 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.458 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.458 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.458 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.458 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.458 INFO:tasks.ceph:[global] mon client directed command retry = 5
2024-05-08T01:13:55.458 INFO:tasks.ceph:[global] mon warn on pool no app = False
2024-05-08T01:13:55.458 INFO:tasks.ceph:[global] ms inject socket failures = 5000
2024-05-08T01:13:55.458 INFO:tasks.ceph:[mgr] debug mgr = 20
2024-05-08T01:13:55.459 INFO:tasks.ceph:[mgr] debug ms = 1
2024-05-08T01:13:55.459 INFO:tasks.ceph:[mon] debug mon = 20
2024-05-08T01:13:55.459 INFO:tasks.ceph:[mon] debug ms = 1
2024-05-08T01:13:55.459 INFO:tasks.ceph:[mon] debug paxos = 20
2024-05-08T01:13:55.459 INFO:tasks.ceph:[osd] bluefs allocator = hybrid
2024-05-08T01:13:55.459 INFO:tasks.ceph:[osd] bluestore allocator = hybrid
2024-05-08T01:13:55.459 INFO:tasks.ceph:[osd] bluestore block size = 96636764160
2024-05-08T01:13:55.459 INFO:tasks.ceph:[osd] bluestore fsck on mount = True
2024-05-08T01:13:55.459 INFO:tasks.ceph:[osd] debug bluefs = 1/20
2024-05-08T01:13:55.459 INFO:tasks.ceph:[osd] debug bluestore = 1/20
2024-05-08T01:13:55.459 INFO:tasks.ceph:[osd] debug ms = 1
2024-05-08T01:13:55.459 INFO:tasks.ceph:[osd] debug osd = 20
2024-05-08T01:13:55.459 INFO:tasks.ceph:[osd] debug rocksdb = 4/10
2024-05-08T01:13:55.459 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85
2024-05-08T01:13:55.460 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9
2024-05-08T01:13:55.460 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8
2024-05-08T01:13:55.460 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95
2024-05-08T01:13:55.460 INFO:tasks.ceph:[osd] osd objectstore = bluestore
2024-05-08T01:13:55.460 INFO:tasks.ceph:Setting up cluster2.mon.a...
2024-05-08T01:13:55.460 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring
2024-05-08T01:13:55.529 INFO:teuthology.orchestra.run.smithi142.stdout:creating /etc/ceph/cluster2.keyring
2024-05-08T01:13:55.532 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring
2024-05-08T01:13:55.608 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0644 /etc/ceph/cluster2.keyring
2024-05-08T01:13:55.661 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.142')]
2024-05-08T01:13:55.662 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.142', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}}
2024-05-08T01:13:55.663 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:55.663 DEBUG:teuthology.orchestra.run.smithi142:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf
2024-05-08T01:13:55.707 DEBUG:teuthology.orchestra.run.smithi142:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.142 --print /home/ubuntu/cephtest/cluster2.monmap
2024-05-08T01:13:55.785 INFO:teuthology.orchestra.run.smithi142.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap
2024-05-08T01:13:55.785 INFO:teuthology.orchestra.run.smithi142.stdout:monmaptool: generated fsid e7480526-62e8-4ed0-8efe-58a8e36c6942
2024-05-08T01:13:55.785 INFO:teuthology.orchestra.run.smithi142.stdout:setting min_mon_release = quincy
2024-05-08T01:13:55.785 INFO:teuthology.orchestra.run.smithi142.stdout:epoch 0
2024-05-08T01:13:55.786 INFO:teuthology.orchestra.run.smithi142.stdout:fsid e7480526-62e8-4ed0-8efe-58a8e36c6942
2024-05-08T01:13:55.786 INFO:teuthology.orchestra.run.smithi142.stdout:last_changed 2024-05-08T01:13:55.807214+0000
2024-05-08T01:13:55.786 INFO:teuthology.orchestra.run.smithi142.stdout:created 2024-05-08T01:13:55.807214+0000
2024-05-08T01:13:55.786 INFO:teuthology.orchestra.run.smithi142.stdout:min_mon_release 17 (quincy)
2024-05-08T01:13:55.786 INFO:teuthology.orchestra.run.smithi142.stdout:election_strategy: 1
2024-05-08T01:13:55.786 INFO:teuthology.orchestra.run.smithi142.stdout:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a
2024-05-08T01:13:55.786 INFO:teuthology.orchestra.run.smithi142.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors)
2024-05-08T01:13:55.787 DEBUG:teuthology.orchestra.run.smithi142:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf
2024-05-08T01:13:55.835 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID e7480526-62e8-4ed0-8efe-58a8e36c6942...
2024-05-08T01:13:55.836 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null
2024-05-08T01:13:55.839 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null
2024-05-08T01:13:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:[global]
2024-05-08T01:13:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:	chdir = ""
2024-05-08T01:13:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:	pid file = /var/run/ceph/$cluster-$name.pid
2024-05-08T01:13:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:	auth supported = cephx
2024-05-08T01:13:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:	filestore xattr use omap = true
2024-05-08T01:13:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:	mon clock drift allowed = 1.000
2024-05-08T01:13:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:	osd crush chooseleaf type = 0
2024-05-08T01:13:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:	auth debug = true
2024-05-08T01:13:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:	ms die on old message = true
2024-05-08T01:13:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:	ms die on bug = true
2024-05-08T01:13:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:	mon max pg per osd = 10000	# >= luminous
2024-05-08T01:13:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:	mon pg warn max object skew = 0
2024-05-08T01:13:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:	# disable pg_autoscaler by default for new pools
2024-05-08T01:13:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:	osd_pool_default_pg_autoscale_mode = off
2024-05-08T01:13:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:	osd pool default size = 2
2024-05-08T01:13:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:	mon osd allow primary affinity = true
2024-05-08T01:13:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:	mon osd allow pg remap = true
2024-05-08T01:13:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:	mon warn on legacy crush tunables = false
2024-05-08T01:13:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:	mon warn on crush straw calc version zero = false
2024-05-08T01:13:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:	mon warn on no sortbitwise = false
2024-05-08T01:13:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:	mon warn on osd down out interval zero = false
2024-05-08T01:13:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:	mon warn on too few osds = false
2024-05-08T01:13:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:	mon_warn_on_pool_pg_num_not_power_of_two = false
2024-05-08T01:13:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:	mon_warn_on_pool_no_redundancy = false
2024-05-08T01:13:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:	mon_allow_pool_size_one = true
2024-05-08T01:13:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:	osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd
2024-05-08T01:13:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:	osd default data pool replay window = 5
2024-05-08T01:13:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:	mon allow pool delete = true
2024-05-08T01:13:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:	mon cluster log file level = debug
2024-05-08T01:13:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:	debug asserts on shutdown = true
2024-05-08T01:13:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:	mon health detail to clog = false
2024-05-08T01:13:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:	mon host = 172.21.15.142
2024-05-08T01:13:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:	mon client directed command retry = 5
2024-05-08T01:13:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:	mon warn on pool no app = False
2024-05-08T01:13:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:	ms inject socket failures = 5000
2024-05-08T01:13:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:	fsid = e7480526-62e8-4ed0-8efe-58a8e36c6942
2024-05-08T01:13:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:[osd]
2024-05-08T01:13:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:	osd journal size = 100
2024-05-08T01:13:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:	osd scrub load threshold = 5.0
2024-05-08T01:13:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:	osd scrub max interval = 600
2024-05-08T01:13:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:	osd mclock profile = high_recovery_ops
2024-05-08T01:13:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:	osd recover clone overlap = true
2024-05-08T01:13:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:	osd recovery max chunk = 1048576
2024-05-08T01:13:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:	osd debug shutdown = true
2024-05-08T01:13:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:	osd debug op order = true
2024-05-08T01:13:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:	osd debug verify stray on activate = true
2024-05-08T01:13:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:	osd debug trim objects = true
2024-05-08T01:13:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:	osd open classes on start = true
2024-05-08T01:13:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:	osd debug pg log writeout = true
2024-05-08T01:13:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:	osd deep scrub update digest min age = 30
2024-05-08T01:13:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:	osd map max advance = 10
2024-05-08T01:13:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:	journal zero on create = true
2024-05-08T01:13:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:	filestore ondisk finisher threads = 3
2024-05-08T01:13:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:	filestore apply finisher threads = 3
2024-05-08T01:13:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:	bdev debug aio = true
2024-05-08T01:13:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:	osd debug misdirected ops = true
2024-05-08T01:13:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:	bluefs allocator = hybrid
2024-05-08T01:13:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:	bluestore allocator = hybrid
2024-05-08T01:13:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:	bluestore block size = 96636764160
2024-05-08T01:13:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:	bluestore fsck on mount = True
2024-05-08T01:13:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:	debug bluefs = 1/20
2024-05-08T01:13:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:	debug bluestore = 1/20
2024-05-08T01:13:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:	debug ms = 1
2024-05-08T01:13:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:	debug osd = 20
2024-05-08T01:13:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:	debug rocksdb = 4/10
2024-05-08T01:13:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:	mon osd backfillfull_ratio = 0.85
2024-05-08T01:13:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:	mon osd full ratio = 0.9
2024-05-08T01:13:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:	mon osd nearfull ratio = 0.8
2024-05-08T01:13:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:	osd failsafe full ratio = 0.95
2024-05-08T01:13:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:	osd objectstore = bluestore
2024-05-08T01:13:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:[mgr]
2024-05-08T01:13:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:	debug ms = 1
2024-05-08T01:13:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:	debug mgr = 20
2024-05-08T01:13:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:	debug mon = 20
2024-05-08T01:13:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:	debug auth = 20
2024-05-08T01:13:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:	mon reweight min pgs per osd = 4
2024-05-08T01:13:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:	mon reweight min bytes per osd = 10
2024-05-08T01:13:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:	mgr/telemetry/nag = false
2024-05-08T01:13:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:[mon]
2024-05-08T01:13:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:	debug ms = 1
2024-05-08T01:13:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:	debug mon = 20
2024-05-08T01:13:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:	debug paxos = 20
2024-05-08T01:13:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:	debug auth = 20
2024-05-08T01:13:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:	mon data avail warn = 5
2024-05-08T01:13:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:	mon mgr mkfs grace = 240
2024-05-08T01:13:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:	mon reweight min pgs per osd = 4
2024-05-08T01:13:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:	mon osd reporter subtree level = osd
2024-05-08T01:13:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:	mon osd prime pg temp = true
2024-05-08T01:13:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:	mon reweight min bytes per osd = 10
2024-05-08T01:13:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:	# rotate auth tickets quickly to exercise renewal paths
2024-05-08T01:13:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:	auth mon ticket ttl = 660	# 11m
2024-05-08T01:13:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:	auth service ticket ttl = 240	# 4m
2024-05-08T01:13:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:	# don't complain about insecure global_id in the test suite
2024-05-08T01:13:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:	mon_warn_on_insecure_global_id_reclaim = false
2024-05-08T01:13:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:	mon_warn_on_insecure_global_id_reclaim_allowed = false
2024-05-08T01:13:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:	# 1m isn't quite enough
2024-05-08T01:13:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:	mon_down_mkfs_grace = 2m
2024-05-08T01:13:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:	mon_warn_on_filestore_osds = false
2024-05-08T01:13:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:13:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:[client]
2024-05-08T01:13:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:	rgw cache enabled = true
2024-05-08T01:13:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:	rgw enable ops log = true
2024-05-08T01:13:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:	rgw enable usage log = true
2024-05-08T01:13:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:	log file = /var/log/ceph/$cluster-$name.$pid.log
2024-05-08T01:13:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:	admin socket = /var/run/ceph/$cluster-$name.$pid.asok
2024-05-08T01:13:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:	debug rbd = 20
2024-05-08T01:13:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:	debug rbd_mirror = 15
2024-05-08T01:13:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:	log to stderr = False
2024-05-08T01:13:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:	rbd default features = 125
2024-05-08T01:13:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:	rbd mirror image policy type = none
2024-05-08T01:13:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:[mon.a]
2024-05-08T01:13:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.0]
2024-05-08T01:13:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.1]
2024-05-08T01:13:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.2]
2024-05-08T01:13:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.3]
2024-05-08T01:13:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.4]
2024-05-08T01:13:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.893 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.5]
2024-05-08T01:13:55.893 INFO:teuthology.orchestra.run.smithi106.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.893 INFO:teuthology.orchestra.run.smithi106.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.893 INFO:teuthology.orchestra.run.smithi106.stdout:[client.mirror.6]
2024-05-08T01:13:55.893 INFO:teuthology.orchestra.run.smithi106.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.893 INFO:teuthology.orchestra.run.smithi106.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.911 INFO:teuthology.orchestra.run.smithi142.stdout:[global]
2024-05-08T01:13:55.911 INFO:teuthology.orchestra.run.smithi142.stdout:	chdir = ""
2024-05-08T01:13:55.911 INFO:teuthology.orchestra.run.smithi142.stdout:	pid file = /var/run/ceph/$cluster-$name.pid
2024-05-08T01:13:55.912 INFO:teuthology.orchestra.run.smithi142.stdout:	auth supported = cephx
2024-05-08T01:13:55.912 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.912 INFO:teuthology.orchestra.run.smithi142.stdout:	filestore xattr use omap = true
2024-05-08T01:13:55.912 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.912 INFO:teuthology.orchestra.run.smithi142.stdout:	mon clock drift allowed = 1.000
2024-05-08T01:13:55.912 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.912 INFO:teuthology.orchestra.run.smithi142.stdout:	osd crush chooseleaf type = 0
2024-05-08T01:13:55.912 INFO:teuthology.orchestra.run.smithi142.stdout:	auth debug = true
2024-05-08T01:13:55.912 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.912 INFO:teuthology.orchestra.run.smithi142.stdout:	ms die on old message = true
2024-05-08T01:13:55.912 INFO:teuthology.orchestra.run.smithi142.stdout:	ms die on bug = true
2024-05-08T01:13:55.912 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.912 INFO:teuthology.orchestra.run.smithi142.stdout:	mon max pg per osd = 10000	# >= luminous
2024-05-08T01:13:55.912 INFO:teuthology.orchestra.run.smithi142.stdout:	mon pg warn max object skew = 0
2024-05-08T01:13:55.912 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.913 INFO:teuthology.orchestra.run.smithi142.stdout:	# disable pg_autoscaler by default for new pools
2024-05-08T01:13:55.913 INFO:teuthology.orchestra.run.smithi142.stdout:	osd_pool_default_pg_autoscale_mode = off
2024-05-08T01:13:55.913 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.913 INFO:teuthology.orchestra.run.smithi142.stdout:	osd pool default size = 2
2024-05-08T01:13:55.913 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.913 INFO:teuthology.orchestra.run.smithi142.stdout:	mon osd allow primary affinity = true
2024-05-08T01:13:55.913 INFO:teuthology.orchestra.run.smithi142.stdout:	mon osd allow pg remap = true
2024-05-08T01:13:55.913 INFO:teuthology.orchestra.run.smithi142.stdout:	mon warn on legacy crush tunables = false
2024-05-08T01:13:55.913 INFO:teuthology.orchestra.run.smithi142.stdout:	mon warn on crush straw calc version zero = false
2024-05-08T01:13:55.913 INFO:teuthology.orchestra.run.smithi142.stdout:	mon warn on no sortbitwise = false
2024-05-08T01:13:55.913 INFO:teuthology.orchestra.run.smithi142.stdout:	mon warn on osd down out interval zero = false
2024-05-08T01:13:55.913 INFO:teuthology.orchestra.run.smithi142.stdout:	mon warn on too few osds = false
2024-05-08T01:13:55.914 INFO:teuthology.orchestra.run.smithi142.stdout:	mon_warn_on_pool_pg_num_not_power_of_two = false
2024-05-08T01:13:55.914 INFO:teuthology.orchestra.run.smithi142.stdout:	mon_warn_on_pool_no_redundancy = false
2024-05-08T01:13:55.914 INFO:teuthology.orchestra.run.smithi142.stdout:	mon_allow_pool_size_one = true
2024-05-08T01:13:55.914 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.914 INFO:teuthology.orchestra.run.smithi142.stdout:	osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd
2024-05-08T01:13:55.914 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.914 INFO:teuthology.orchestra.run.smithi142.stdout:	osd default data pool replay window = 5
2024-05-08T01:13:55.914 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.914 INFO:teuthology.orchestra.run.smithi142.stdout:	mon allow pool delete = true
2024-05-08T01:13:55.914 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.914 INFO:teuthology.orchestra.run.smithi142.stdout:	mon cluster log file level = debug
2024-05-08T01:13:55.914 INFO:teuthology.orchestra.run.smithi142.stdout:	debug asserts on shutdown = true
2024-05-08T01:13:55.914 INFO:teuthology.orchestra.run.smithi142.stdout:	mon health detail to clog = false
2024-05-08T01:13:55.915 INFO:teuthology.orchestra.run.smithi142.stdout:	mon host = 172.21.15.142
2024-05-08T01:13:55.915 INFO:teuthology.orchestra.run.smithi142.stdout:	mon client directed command retry = 5
2024-05-08T01:13:55.915 INFO:teuthology.orchestra.run.smithi142.stdout:	mon warn on pool no app = False
2024-05-08T01:13:55.915 INFO:teuthology.orchestra.run.smithi142.stdout:	ms inject socket failures = 5000
2024-05-08T01:13:55.915 INFO:teuthology.orchestra.run.smithi142.stdout:	fsid = e7480526-62e8-4ed0-8efe-58a8e36c6942
2024-05-08T01:13:55.915 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.915 INFO:teuthology.orchestra.run.smithi142.stdout:[osd]
2024-05-08T01:13:55.915 INFO:teuthology.orchestra.run.smithi142.stdout:	osd journal size = 100
2024-05-08T01:13:55.915 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.915 INFO:teuthology.orchestra.run.smithi142.stdout:	osd scrub load threshold = 5.0
2024-05-08T01:13:55.915 INFO:teuthology.orchestra.run.smithi142.stdout:	osd scrub max interval = 600
2024-05-08T01:13:55.915 INFO:teuthology.orchestra.run.smithi142.stdout:	osd mclock profile = high_recovery_ops
2024-05-08T01:13:55.915 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.915 INFO:teuthology.orchestra.run.smithi142.stdout:	osd recover clone overlap = true
2024-05-08T01:13:55.915 INFO:teuthology.orchestra.run.smithi142.stdout:	osd recovery max chunk = 1048576
2024-05-08T01:13:55.916 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.916 INFO:teuthology.orchestra.run.smithi142.stdout:	osd debug shutdown = true
2024-05-08T01:13:55.916 INFO:teuthology.orchestra.run.smithi142.stdout:	osd debug op order = true
2024-05-08T01:13:55.916 INFO:teuthology.orchestra.run.smithi142.stdout:	osd debug verify stray on activate = true
2024-05-08T01:13:55.916 INFO:teuthology.orchestra.run.smithi142.stdout:	osd debug trim objects = true
2024-05-08T01:13:55.916 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.916 INFO:teuthology.orchestra.run.smithi142.stdout:	osd open classes on start = true
2024-05-08T01:13:55.916 INFO:teuthology.orchestra.run.smithi142.stdout:	osd debug pg log writeout = true
2024-05-08T01:13:55.916 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.916 INFO:teuthology.orchestra.run.smithi142.stdout:	osd deep scrub update digest min age = 30
2024-05-08T01:13:55.916 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.916 INFO:teuthology.orchestra.run.smithi142.stdout:	osd map max advance = 10
2024-05-08T01:13:55.916 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.916 INFO:teuthology.orchestra.run.smithi142.stdout:	journal zero on create = true
2024-05-08T01:13:55.917 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.917 INFO:teuthology.orchestra.run.smithi142.stdout:	filestore ondisk finisher threads = 3
2024-05-08T01:13:55.917 INFO:teuthology.orchestra.run.smithi142.stdout:	filestore apply finisher threads = 3
2024-05-08T01:13:55.917 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.917 INFO:teuthology.orchestra.run.smithi142.stdout:	bdev debug aio = true
2024-05-08T01:13:55.917 INFO:teuthology.orchestra.run.smithi142.stdout:	osd debug misdirected ops = true
2024-05-08T01:13:55.917 INFO:teuthology.orchestra.run.smithi142.stdout:	bluefs allocator = hybrid
2024-05-08T01:13:55.917 INFO:teuthology.orchestra.run.smithi142.stdout:	bluestore allocator = hybrid
2024-05-08T01:13:55.917 INFO:teuthology.orchestra.run.smithi142.stdout:	bluestore block size = 96636764160
2024-05-08T01:13:55.917 INFO:teuthology.orchestra.run.smithi142.stdout:	bluestore fsck on mount = True
2024-05-08T01:13:55.917 INFO:teuthology.orchestra.run.smithi142.stdout:	debug bluefs = 1/20
2024-05-08T01:13:55.917 INFO:teuthology.orchestra.run.smithi142.stdout:	debug bluestore = 1/20
2024-05-08T01:13:55.917 INFO:teuthology.orchestra.run.smithi142.stdout:	debug ms = 1
2024-05-08T01:13:55.917 INFO:teuthology.orchestra.run.smithi142.stdout:	debug osd = 20
2024-05-08T01:13:55.918 INFO:teuthology.orchestra.run.smithi142.stdout:	debug rocksdb = 4/10
2024-05-08T01:13:55.918 INFO:teuthology.orchestra.run.smithi142.stdout:	mon osd backfillfull_ratio = 0.85
2024-05-08T01:13:55.918 INFO:teuthology.orchestra.run.smithi142.stdout:	mon osd full ratio = 0.9
2024-05-08T01:13:55.918 INFO:teuthology.orchestra.run.smithi142.stdout:	mon osd nearfull ratio = 0.8
2024-05-08T01:13:55.918 INFO:teuthology.orchestra.run.smithi142.stdout:	osd failsafe full ratio = 0.95
2024-05-08T01:13:55.918 INFO:teuthology.orchestra.run.smithi142.stdout:	osd objectstore = bluestore
2024-05-08T01:13:55.918 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.918 INFO:teuthology.orchestra.run.smithi142.stdout:[mgr]
2024-05-08T01:13:55.918 INFO:teuthology.orchestra.run.smithi142.stdout:	debug ms = 1
2024-05-08T01:13:55.918 INFO:teuthology.orchestra.run.smithi142.stdout:	debug mgr = 20
2024-05-08T01:13:55.918 INFO:teuthology.orchestra.run.smithi142.stdout:	debug mon = 20
2024-05-08T01:13:55.918 INFO:teuthology.orchestra.run.smithi142.stdout:	debug auth = 20
2024-05-08T01:13:55.918 INFO:teuthology.orchestra.run.smithi142.stdout:	mon reweight min pgs per osd = 4
2024-05-08T01:13:55.918 INFO:teuthology.orchestra.run.smithi142.stdout:	mon reweight min bytes per osd = 10
2024-05-08T01:13:55.919 INFO:teuthology.orchestra.run.smithi142.stdout:	mgr/telemetry/nag = false
2024-05-08T01:13:55.919 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.919 INFO:teuthology.orchestra.run.smithi142.stdout:[mon]
2024-05-08T01:13:55.919 INFO:teuthology.orchestra.run.smithi142.stdout:	debug ms = 1
2024-05-08T01:13:55.919 INFO:teuthology.orchestra.run.smithi142.stdout:	debug mon = 20
2024-05-08T01:13:55.919 INFO:teuthology.orchestra.run.smithi142.stdout:	debug paxos = 20
2024-05-08T01:13:55.919 INFO:teuthology.orchestra.run.smithi142.stdout:	debug auth = 20
2024-05-08T01:13:55.919 INFO:teuthology.orchestra.run.smithi142.stdout:	mon data avail warn = 5
2024-05-08T01:13:55.919 INFO:teuthology.orchestra.run.smithi142.stdout:	mon mgr mkfs grace = 240
2024-05-08T01:13:55.919 INFO:teuthology.orchestra.run.smithi142.stdout:	mon reweight min pgs per osd = 4
2024-05-08T01:13:55.919 INFO:teuthology.orchestra.run.smithi142.stdout:	mon osd reporter subtree level = osd
2024-05-08T01:13:55.919 INFO:teuthology.orchestra.run.smithi142.stdout:	mon osd prime pg temp = true
2024-05-08T01:13:55.919 INFO:teuthology.orchestra.run.smithi142.stdout:	mon reweight min bytes per osd = 10
2024-05-08T01:13:55.919 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.920 INFO:teuthology.orchestra.run.smithi142.stdout:	# rotate auth tickets quickly to exercise renewal paths
2024-05-08T01:13:55.920 INFO:teuthology.orchestra.run.smithi142.stdout:	auth mon ticket ttl = 660	# 11m
2024-05-08T01:13:55.920 INFO:teuthology.orchestra.run.smithi142.stdout:	auth service ticket ttl = 240	# 4m
2024-05-08T01:13:55.920 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.920 INFO:teuthology.orchestra.run.smithi142.stdout:	# don't complain about insecure global_id in the test suite
2024-05-08T01:13:55.920 INFO:teuthology.orchestra.run.smithi142.stdout:	mon_warn_on_insecure_global_id_reclaim = false
2024-05-08T01:13:55.920 INFO:teuthology.orchestra.run.smithi142.stdout:	mon_warn_on_insecure_global_id_reclaim_allowed = false
2024-05-08T01:13:55.920 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.920 INFO:teuthology.orchestra.run.smithi142.stdout:	# 1m isn't quite enough
2024-05-08T01:13:55.920 INFO:teuthology.orchestra.run.smithi142.stdout:	mon_down_mkfs_grace = 2m
2024-05-08T01:13:55.920 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.920 INFO:teuthology.orchestra.run.smithi142.stdout:	mon_warn_on_filestore_osds = false
2024-05-08T01:13:55.920 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:13:55.920 INFO:teuthology.orchestra.run.smithi142.stdout:[client]
2024-05-08T01:13:55.920 INFO:teuthology.orchestra.run.smithi142.stdout:	rgw cache enabled = true
2024-05-08T01:13:55.921 INFO:teuthology.orchestra.run.smithi142.stdout:	rgw enable ops log = true
2024-05-08T01:13:55.921 INFO:teuthology.orchestra.run.smithi142.stdout:	rgw enable usage log = true
2024-05-08T01:13:55.921 INFO:teuthology.orchestra.run.smithi142.stdout:	log file = /var/log/ceph/$cluster-$name.$pid.log
2024-05-08T01:13:55.921 INFO:teuthology.orchestra.run.smithi142.stdout:	admin socket = /var/run/ceph/$cluster-$name.$pid.asok
2024-05-08T01:13:55.921 INFO:teuthology.orchestra.run.smithi142.stdout:	debug rbd = 20
2024-05-08T01:13:55.921 INFO:teuthology.orchestra.run.smithi142.stdout:	debug rbd_mirror = 15
2024-05-08T01:13:55.921 INFO:teuthology.orchestra.run.smithi142.stdout:	log to stderr = False
2024-05-08T01:13:55.921 INFO:teuthology.orchestra.run.smithi142.stdout:	rbd default features = 125
2024-05-08T01:13:55.921 INFO:teuthology.orchestra.run.smithi142.stdout:	rbd mirror image policy type = none
2024-05-08T01:13:55.921 INFO:teuthology.orchestra.run.smithi142.stdout:[mon.a]
2024-05-08T01:13:55.921 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.0]
2024-05-08T01:13:55.921 INFO:teuthology.orchestra.run.smithi142.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.921 INFO:teuthology.orchestra.run.smithi142.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.921 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.1]
2024-05-08T01:13:55.921 INFO:teuthology.orchestra.run.smithi142.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.922 INFO:teuthology.orchestra.run.smithi142.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.922 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.2]
2024-05-08T01:13:55.922 INFO:teuthology.orchestra.run.smithi142.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.922 INFO:teuthology.orchestra.run.smithi142.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.922 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.3]
2024-05-08T01:13:55.922 INFO:teuthology.orchestra.run.smithi142.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.922 INFO:teuthology.orchestra.run.smithi142.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.922 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.4]
2024-05-08T01:13:55.922 INFO:teuthology.orchestra.run.smithi142.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.922 INFO:teuthology.orchestra.run.smithi142.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.922 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.5]
2024-05-08T01:13:55.922 INFO:teuthology.orchestra.run.smithi142.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.922 INFO:teuthology.orchestra.run.smithi142.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.922 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.6]
2024-05-08T01:13:55.923 INFO:teuthology.orchestra.run.smithi142.stdout:	admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok
2024-05-08T01:13:55.923 INFO:teuthology.orchestra.run.smithi142.stdout:	pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid
2024-05-08T01:13:55.924 INFO:tasks.ceph:Creating admin key on cluster2.mon.a...
2024-05-08T01:13:55.924 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring
2024-05-08T01:13:56.016 INFO:tasks.ceph:Copying monmap to all nodes...
2024-05-08T01:13:56.016 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:56.016 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout
2024-05-08T01:13:56.021 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:56.021 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout
2024-05-08T01:13:56.066 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi106.front.sepia.ceph.com
2024-05-08T01:13:56.067 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:56.067 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/cluster2.keyring
2024-05-08T01:13:56.067 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0644 /etc/ceph/cluster2.keyring
2024-05-08T01:13:56.091 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:13:56.091 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/home/ubuntu/cephtest/cluster2.monmap
2024-05-08T01:13:56.138 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi142.front.sepia.ceph.com
2024-05-08T01:13:56.138 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:56.138 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/cluster2.keyring
2024-05-08T01:13:56.138 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0644 /etc/ceph/cluster2.keyring
2024-05-08T01:13:56.162 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:13:56.162 DEBUG:teuthology.orchestra.run.smithi142:> dd of=/home/ubuntu/cephtest/cluster2.monmap
2024-05-08T01:13:56.211 INFO:tasks.ceph:Setting up mon nodes...
2024-05-08T01:13:56.211 INFO:tasks.ceph:Setting up mgr nodes...
2024-05-08T01:13:56.211 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring
2024-05-08T01:13:56.254 INFO:teuthology.orchestra.run.smithi106.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring
2024-05-08T01:13:56.257 INFO:tasks.ceph:Setting up mds nodes...
2024-05-08T01:13:56.257 INFO:tasks.ceph_client:Setting up client nodes...
2024-05-08T01:13:56.258 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring
2024-05-08T01:13:56.334 INFO:teuthology.orchestra.run.smithi106.stdout:creating /etc/ceph/cluster2.client.0.keyring
2024-05-08T01:13:56.343 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring
2024-05-08T01:13:56.379 INFO:teuthology.orchestra.run.smithi142.stdout:creating /etc/ceph/cluster2.client.mirror.keyring
2024-05-08T01:13:56.390 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring
2024-05-08T01:13:56.463 INFO:teuthology.orchestra.run.smithi142.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring
2024-05-08T01:13:56.474 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring
2024-05-08T01:13:56.554 INFO:teuthology.orchestra.run.smithi142.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring
2024-05-08T01:13:56.564 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring
2024-05-08T01:13:56.638 INFO:teuthology.orchestra.run.smithi142.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring
2024-05-08T01:13:56.648 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring
2024-05-08T01:13:56.726 INFO:teuthology.orchestra.run.smithi142.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring
2024-05-08T01:13:56.737 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring
2024-05-08T01:13:56.811 INFO:teuthology.orchestra.run.smithi142.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring
2024-05-08T01:13:56.822 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring
2024-05-08T01:13:56.903 INFO:teuthology.orchestra.run.smithi142.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring
2024-05-08T01:13:56.913 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring
2024-05-08T01:13:56.985 INFO:teuthology.orchestra.run.smithi142.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring
2024-05-08T01:13:56.996 INFO:tasks.ceph:Running mkfs on osd nodes...
2024-05-08T01:13:56.997 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi106.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi142.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}}
2024-05-08T01:13:56.997 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0
2024-05-08T01:13:57.050 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}
2024-05-08T01:13:57.051 INFO:tasks.ceph:role: cluster2.osd.0
2024-05-08T01:13:57.051 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi142.front.sepia.ceph.com
2024-05-08T01:13:57.051 DEBUG:teuthology.orchestra.run.smithi142:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1
2024-05-08T01:13:57.112 INFO:teuthology.orchestra.run.smithi142.stdout:meta-data=/dev/vg_nvme/lv_1      isize=2048   agcount=4, agsize=5859072 blks
2024-05-08T01:13:57.113 INFO:teuthology.orchestra.run.smithi142.stdout:         =                       sectsz=512   attr=2, projid32bit=1
2024-05-08T01:13:57.113 INFO:teuthology.orchestra.run.smithi142.stdout:         =                       crc=1        finobt=1, sparse=1, rmapbt=0
2024-05-08T01:13:57.113 INFO:teuthology.orchestra.run.smithi142.stdout:         =                       reflink=1    bigtime=0 inobtcount=0
2024-05-08T01:13:57.113 INFO:teuthology.orchestra.run.smithi142.stdout:data     =                       bsize=4096   blocks=23436288, imaxpct=25
2024-05-08T01:13:57.113 INFO:teuthology.orchestra.run.smithi142.stdout:         =                       sunit=0      swidth=0 blks
2024-05-08T01:13:57.113 INFO:teuthology.orchestra.run.smithi142.stdout:naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
2024-05-08T01:13:57.113 INFO:teuthology.orchestra.run.smithi142.stdout:log      =internal log           bsize=4096   blocks=11443, version=2
2024-05-08T01:13:57.113 INFO:teuthology.orchestra.run.smithi142.stdout:         =                       sectsz=512   sunit=0 blks, lazy-count=1
2024-05-08T01:13:57.113 INFO:teuthology.orchestra.run.smithi142.stdout:realtime =none                   extsz=4096   blocks=0, rtextents=0
2024-05-08T01:13:57.597 INFO:teuthology.orchestra.run.smithi142.stdout:Discarding blocks...Done.
2024-05-08T01:13:57.599 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi142.front.sepia.ceph.com -o noatime
2024-05-08T01:13:57.599 DEBUG:teuthology.orchestra.run.smithi142:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0
2024-05-08T01:13:57.765 DEBUG:teuthology.orchestra.run.smithi142:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0
2024-05-08T01:13:57.775 INFO:teuthology.orchestra.run.smithi142.stderr:sudo: /sbin/restorecon: command not found
2024-05-08T01:13:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T01:13:57.776 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1
2024-05-08T01:13:57.831 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}
2024-05-08T01:13:57.831 INFO:tasks.ceph:role: cluster2.osd.1
2024-05-08T01:13:57.831 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi142.front.sepia.ceph.com
2024-05-08T01:13:57.831 DEBUG:teuthology.orchestra.run.smithi142:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2
2024-05-08T01:13:57.894 INFO:teuthology.orchestra.run.smithi142.stdout:meta-data=/dev/vg_nvme/lv_2      isize=2048   agcount=4, agsize=5859072 blks
2024-05-08T01:13:57.895 INFO:teuthology.orchestra.run.smithi142.stdout:         =                       sectsz=512   attr=2, projid32bit=1
2024-05-08T01:13:57.895 INFO:teuthology.orchestra.run.smithi142.stdout:         =                       crc=1        finobt=1, sparse=1, rmapbt=0
2024-05-08T01:13:57.895 INFO:teuthology.orchestra.run.smithi142.stdout:         =                       reflink=1    bigtime=0 inobtcount=0
2024-05-08T01:13:57.895 INFO:teuthology.orchestra.run.smithi142.stdout:data     =                       bsize=4096   blocks=23436288, imaxpct=25
2024-05-08T01:13:57.895 INFO:teuthology.orchestra.run.smithi142.stdout:         =                       sunit=0      swidth=0 blks
2024-05-08T01:13:57.895 INFO:teuthology.orchestra.run.smithi142.stdout:naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
2024-05-08T01:13:57.895 INFO:teuthology.orchestra.run.smithi142.stdout:log      =internal log           bsize=4096   blocks=11443, version=2
2024-05-08T01:13:57.895 INFO:teuthology.orchestra.run.smithi142.stdout:         =                       sectsz=512   sunit=0 blks, lazy-count=1
2024-05-08T01:13:57.895 INFO:teuthology.orchestra.run.smithi142.stdout:realtime =none                   extsz=4096   blocks=0, rtextents=0
2024-05-08T01:13:58.338 INFO:teuthology.orchestra.run.smithi142.stdout:Discarding blocks...Done.
2024-05-08T01:13:58.340 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi142.front.sepia.ceph.com -o noatime
2024-05-08T01:13:58.340 DEBUG:teuthology.orchestra.run.smithi142:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1
2024-05-08T01:13:58.418 DEBUG:teuthology.orchestra.run.smithi142:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1
2024-05-08T01:13:58.469 INFO:teuthology.orchestra.run.smithi142.stderr:sudo: /sbin/restorecon: command not found
2024-05-08T01:13:58.469 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T01:13:58.470 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2
2024-05-08T01:13:58.526 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}
2024-05-08T01:13:58.527 INFO:tasks.ceph:role: cluster2.osd.2
2024-05-08T01:13:58.527 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi142.front.sepia.ceph.com
2024-05-08T01:13:58.527 DEBUG:teuthology.orchestra.run.smithi142:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3
2024-05-08T01:13:58.590 INFO:teuthology.orchestra.run.smithi142.stdout:meta-data=/dev/vg_nvme/lv_3      isize=2048   agcount=4, agsize=5859072 blks
2024-05-08T01:13:58.591 INFO:teuthology.orchestra.run.smithi142.stdout:         =                       sectsz=512   attr=2, projid32bit=1
2024-05-08T01:13:58.591 INFO:teuthology.orchestra.run.smithi142.stdout:         =                       crc=1        finobt=1, sparse=1, rmapbt=0
2024-05-08T01:13:58.591 INFO:teuthology.orchestra.run.smithi142.stdout:         =                       reflink=1    bigtime=0 inobtcount=0
2024-05-08T01:13:58.591 INFO:teuthology.orchestra.run.smithi142.stdout:data     =                       bsize=4096   blocks=23436288, imaxpct=25
2024-05-08T01:13:58.591 INFO:teuthology.orchestra.run.smithi142.stdout:         =                       sunit=0      swidth=0 blks
2024-05-08T01:13:58.591 INFO:teuthology.orchestra.run.smithi142.stdout:naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
2024-05-08T01:13:58.591 INFO:teuthology.orchestra.run.smithi142.stdout:log      =internal log           bsize=4096   blocks=11443, version=2
2024-05-08T01:13:58.591 INFO:teuthology.orchestra.run.smithi142.stdout:         =                       sectsz=512   sunit=0 blks, lazy-count=1
2024-05-08T01:13:58.591 INFO:teuthology.orchestra.run.smithi142.stdout:realtime =none                   extsz=4096   blocks=0, rtextents=0
2024-05-08T01:13:59.053 INFO:teuthology.orchestra.run.smithi142.stdout:Discarding blocks...Done.
2024-05-08T01:13:59.055 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi142.front.sepia.ceph.com -o noatime
2024-05-08T01:13:59.055 DEBUG:teuthology.orchestra.run.smithi142:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2
2024-05-08T01:13:59.180 DEBUG:teuthology.orchestra.run.smithi142:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2
2024-05-08T01:13:59.192 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T01:13:59.193 INFO:teuthology.orchestra.run.smithi142.stderr:sudo: /sbin/restorecon: command not found
2024-05-08T01:13:59.193 DEBUG:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap
2024-05-08T01:13:59.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-08T01:13:59.326+0000 7f7b7b3cc7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory
2024-05-08T01:13:59.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-08T01:13:59.326+0000 7f7b7b3cc7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring
2024-05-08T01:13:59.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-08T01:13:59.326+0000 7f7b7b3cc7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory
2024-05-08T01:13:59.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-08T01:13:59.326+0000 7f7b7b3cc7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory
2024-05-08T01:13:59.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-08T01:13:59.326+0000 7f7b7b3cc7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory
2024-05-08T01:13:59.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-08T01:13:59.334+0000 7f7b7b3cc7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid
2024-05-08T01:14:02.285 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0
2024-05-08T01:14:02.299 DEBUG:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap
2024-05-08T01:14:02.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-08T01:14:02.398+0000 7fd29ab7e7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory
2024-05-08T01:14:02.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-08T01:14:02.398+0000 7fd29ab7e7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring
2024-05-08T01:14:02.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-08T01:14:02.398+0000 7fd29ab7e7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory
2024-05-08T01:14:02.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-08T01:14:02.398+0000 7fd29ab7e7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory
2024-05-08T01:14:02.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-08T01:14:02.398+0000 7fd29ab7e7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory
2024-05-08T01:14:02.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-08T01:14:02.402+0000 7fd29ab7e7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid
2024-05-08T01:14:05.393 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1
2024-05-08T01:14:05.452 DEBUG:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap
2024-05-08T01:14:05.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-08T01:14:05.550+0000 7f79e87887c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory
2024-05-08T01:14:05.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-08T01:14:05.550+0000 7f79e87887c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring
2024-05-08T01:14:05.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-08T01:14:05.550+0000 7f79e87887c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory
2024-05-08T01:14:05.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-08T01:14:05.550+0000 7f79e87887c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory
2024-05-08T01:14:05.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-08T01:14:05.550+0000 7f79e87887c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory
2024-05-08T01:14:05.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-05-08T01:14:05.554+0000 7f79e87887c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid
2024-05-08T01:14:08.501 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2
2024-05-08T01:14:08.559 INFO:tasks.ceph:Reading keys from all nodes...
2024-05-08T01:14:08.559 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:14:08.559 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout
2024-05-08T01:14:08.574 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:14:08.574 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout
2024-05-08T01:14:08.614 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:14:08.614 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout
2024-05-08T01:14:08.669 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:14:08.669 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout
2024-05-08T01:14:08.721 DEBUG:teuthology.orchestra.run.smithi106:> set -ex
2024-05-08T01:14:08.721 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout
2024-05-08T01:14:08.727 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:14:08.727 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout
2024-05-08T01:14:08.767 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:14:08.767 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout
2024-05-08T01:14:08.815 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:14:08.815 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout
2024-05-08T01:14:08.863 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:14:08.863 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout
2024-05-08T01:14:08.911 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:14:08.911 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout
2024-05-08T01:14:08.959 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:14:08.959 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout
2024-05-08T01:14:09.007 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:14:09.007 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout
2024-05-08T01:14:09.055 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:14:09.055 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout
2024-05-08T01:14:09.107 INFO:tasks.ceph:Adding keys to all mons...
2024-05-08T01:14:09.107 DEBUG:teuthology.orchestra.run.smithi142:> sudo tee -a /etc/ceph/cluster2.keyring
2024-05-08T01:14:09.161 INFO:teuthology.orchestra.run.smithi142.stdout:[mgr.x]
2024-05-08T01:14:09.161 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDU0TpmQ0qLEBAA4lBqSHVpzBoGJBO5lQkS3g==
2024-05-08T01:14:09.161 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.0]
2024-05-08T01:14:09.161 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDX0Tpm90mjExAAQNB6w1HBuE4Qtyp2I2/BcA==
2024-05-08T01:14:09.162 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.1]
2024-05-08T01:14:09.162 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDa0Tpmv4IGGBAADmYzgy9HFSPw3I0quEdpiA==
2024-05-08T01:14:09.162 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.2]
2024-05-08T01:14:09.162 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDd0TpmA9oXIRAABg7d2B5Cyqf2ICAXzzf77Q==
2024-05-08T01:14:09.162 INFO:teuthology.orchestra.run.smithi142.stdout:[client.0]
2024-05-08T01:14:09.162 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDU0TpmuSpAFRAA+ZXp+XFo9cPzvxv6a+aCPg==
2024-05-08T01:14:09.162 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror]
2024-05-08T01:14:09.162 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDU0Tpm6jvlFxAAG4eHE76fsVI+LBp4dAWL/w==
2024-05-08T01:14:09.162 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.0]
2024-05-08T01:14:09.162 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDU0TpmPyrvHBAAw8aH7W5vIanto3wgH7uMkA==
2024-05-08T01:14:09.162 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.1]
2024-05-08T01:14:09.162 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDU0TpmB3RSIhAAAG/lLDbxOoO6j3AemEUzjA==
2024-05-08T01:14:09.162 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.2]
2024-05-08T01:14:09.163 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDU0TpmFPtZJxAAOQPBuOoufjGuEmNq37QuOg==
2024-05-08T01:14:09.163 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.3]
2024-05-08T01:14:09.163 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDU0TpmdeiXLBAAZ28sZWUQCPZ3+cDUY82DrA==
2024-05-08T01:14:09.163 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.4]
2024-05-08T01:14:09.163 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDU0Tpm4uyjMRAAcxjG1afFqDAjGw/SeACdDQ==
2024-05-08T01:14:09.163 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.5]
2024-05-08T01:14:09.163 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDU0TpmGYIeNxAAZ2F9iLCpLGyGqyuT6HE3lg==
2024-05-08T01:14:09.163 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.6]
2024-05-08T01:14:09.163 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDV0TpmzzlzABAAI23N3VUmx57gjGU5fk5zsg==
2024-05-08T01:14:09.164 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
2024-05-08T01:14:09.252 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *'
2024-05-08T01:14:09.293 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *'
2024-05-08T01:14:09.375 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *'
2024-05-08T01:14:09.463 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow
2024-05-08T01:14:09.508 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow
2024-05-08T01:14:09.591 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow
2024-05-08T01:14:09.679 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow
2024-05-08T01:14:09.764 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow
2024-05-08T01:14:09.848 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow
2024-05-08T01:14:09.933 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow
2024-05-08T01:14:10.026 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow
2024-05-08T01:14:10.070 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow
2024-05-08T01:14:10.157 INFO:tasks.ceph:Running mkfs on mon nodes...
2024-05-08T01:14:10.157 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a
2024-05-08T01:14:10.210 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring
2024-05-08T01:14:10.310 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a
2024-05-08T01:14:10.322 DEBUG:teuthology.orchestra.run.smithi142:> rm -- /home/ubuntu/cephtest/cluster2.monmap
2024-05-08T01:14:10.370 INFO:tasks.ceph:Starting mon daemons in cluster cluster2...
2024-05-08T01:14:10.370 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon
2024-05-08T01:14:10.371 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a
2024-05-08T01:14:10.415 INFO:tasks.ceph.cluster2.mon.a:Started
2024-05-08T01:14:10.415 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2...
2024-05-08T01:14:10.416 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon
2024-05-08T01:14:10.416 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x
2024-05-08T01:14:10.418 INFO:tasks.ceph.cluster2.mgr.x:Started
2024-05-08T01:14:10.418 INFO:tasks.ceph:Setting crush tunables to default
2024-05-08T01:14:10.419 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster cluster2 osd crush tunables default
2024-05-08T01:14:10.647 INFO:teuthology.orchestra.run.smithi142.stderr:adjusted tunables profile to default
2024-05-08T01:14:10.666 INFO:tasks.ceph:check_enable_crimson: False
2024-05-08T01:14:10.667 INFO:tasks.ceph:Starting osd daemons in cluster cluster2...
2024-05-08T01:14:10.667 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:14:10.667 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout
2024-05-08T01:14:10.679 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:14:10.679 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout
2024-05-08T01:14:10.680 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:10.696+0000 7f10d227c180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member
2024-05-08T01:14:10.733 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:14:10.733 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout
2024-05-08T01:14:10.785 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster cluster2 osd new 0b037d45-97f6-4d6b-9827-0fae9be5eede 0
2024-05-08T01:14:11.032 INFO:teuthology.orchestra.run.smithi142.stdout:0
2024-05-08T01:14:11.049 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster cluster2 osd new b14aaee3-26c1-43d8-bd53-f04a32fb66de 1
2024-05-08T01:14:11.074 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:11.092+0000 7f10d227c180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member
2024-05-08T01:14:11.202 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:11.220+0000 7f10d227c180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member
2024-05-08T01:14:11.251 INFO:teuthology.orchestra.run.smithi142.stdout:1
2024-05-08T01:14:11.269 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster cluster2 osd new 308ad5c2-db06-434a-8986-73794c7f77e6 2
2024-05-08T01:14:11.471 INFO:teuthology.orchestra.run.smithi142.stdout:2
2024-05-08T01:14:11.488 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon
2024-05-08T01:14:11.489 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0
2024-05-08T01:14:11.491 INFO:tasks.ceph.cluster2.osd.0:Started
2024-05-08T01:14:11.491 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon
2024-05-08T01:14:11.491 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1
2024-05-08T01:14:11.493 INFO:tasks.ceph.cluster2.osd.1:Started
2024-05-08T01:14:11.493 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon
2024-05-08T01:14:11.493 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2
2024-05-08T01:14:11.495 INFO:tasks.ceph.cluster2.osd.2:Started
2024-05-08T01:14:11.495 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json
2024-05-08T01:14:11.610 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:11.628+0000 7f10d227c180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member
2024-05-08T01:14:11.660 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:11.676+0000 7f10d227c180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member
2024-05-08T01:14:11.677 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:14:11.677 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":5,"fsid":"e7480526-62e8-4ed0-8efe-58a8e36c6942","created":"2024-05-08T01:14:10.569210+0000","modified":"2024-05-08T01:14:11.492301+0000","last_up_change":"0.000000","last_in_change":"2024-05-08T01:14:11.492301+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0b037d45-97f6-4d6b-9827-0fae9be5eede","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b14aaee3-26c1-43d8-bd53-f04a32fb66de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"308ad5c2-db06-434a-8986-73794c7f77e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
2024-05-08T01:14:11.695 INFO:tasks.ceph.ceph_manager.cluster2:[]
2024-05-08T01:14:11.695 INFO:tasks.ceph:Waiting for OSDs to come up
2024-05-08T01:14:11.708 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:11.724+0000 7f10d227c180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member
2024-05-08T01:14:11.781 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:11.800+0000 7f10d227c180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member
2024-05-08T01:14:11.830 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:11.848+0000 7f10d227c180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member
2024-05-08T01:14:11.902 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:11.920+0000 7f10d227c180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member
2024-05-08T01:14:11.952 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:11.968+0000 7f10d227c180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member
2024-05-08T01:14:11.997 DEBUG:teuthology.orchestra.run.smithi142:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json
2024-05-08T01:14:12.088 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:12.104+0000 7f10d227c180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member
2024-05-08T01:14:12.100 INFO:tasks.ceph.cluster2.osd.1.smithi142.stderr:2024-05-08T01:14:12.118+0000 7fcd4d74b7c0 -1 Falling back to public interface
2024-05-08T01:14:12.100 INFO:tasks.ceph.cluster2.osd.0.smithi142.stderr:2024-05-08T01:14:12.118+0000 7f7625dd97c0 -1 Falling back to public interface
2024-05-08T01:14:12.100 INFO:tasks.ceph.cluster2.osd.2.smithi142.stderr:2024-05-08T01:14:12.118+0000 7f5007c817c0 -1 Falling back to public interface
2024-05-08T01:14:12.138 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:12.156+0000 7f10d227c180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member
2024-05-08T01:14:12.183 INFO:teuthology.misc.health.smithi142.stdout:
2024-05-08T01:14:12.183 INFO:teuthology.misc.health.smithi142.stdout:{"epoch":5,"fsid":"e7480526-62e8-4ed0-8efe-58a8e36c6942","created":"2024-05-08T01:14:10.569210+0000","modified":"2024-05-08T01:14:11.492301+0000","last_up_change":"0.000000","last_in_change":"2024-05-08T01:14:11.492301+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0b037d45-97f6-4d6b-9827-0fae9be5eede","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b14aaee3-26c1-43d8-bd53-f04a32fb66de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"308ad5c2-db06-434a-8986-73794c7f77e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
2024-05-08T01:14:12.199 DEBUG:teuthology.misc:0 of 3 OSDs are up
2024-05-08T01:14:12.354 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode.
2024-05-08T01:14:12.355 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve.
2024-05-08T01:14:12.355 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:  from numpy import show_config as show_numpy_config
2024-05-08T01:14:12.362 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:12.380+0000 7f10d227c180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member
2024-05-08T01:14:12.493 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:12.508+0000 7f10d227c180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member
2024-05-08T01:14:12.599 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:12.616+0000 7f10d227c180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member
2024-05-08T01:14:12.660 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:12.676+0000 7f10d227c180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member
2024-05-08T01:14:12.791 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:12.808+0000 7f10d227c180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member
2024-05-08T01:14:13.260 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:13.276+0000 7f10d227c180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member
2024-05-08T01:14:13.332 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:13.348+0000 7f10d227c180 -1 mgr[py] Module status has missing NOTIFY_TYPES member
2024-05-08T01:14:13.474 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:13.492+0000 7f10d227c180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member
2024-05-08T01:14:13.545 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:13.564+0000 7f10d227c180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member
2024-05-08T01:14:13.820 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:13.836+0000 7f10d227c180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member
2024-05-08T01:14:13.874 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:13.892+0000 7f10d227c180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member
2024-05-08T01:14:14.889 INFO:tasks.ceph.cluster2.mgr.x.smithi106.stderr:2024-05-08T01:14:14.908+0000 7f10796c1640 -1 mgr.server handle_report got status from non-daemon mon.a
2024-05-08T01:14:15.901 INFO:tasks.ceph.cluster2.osd.1.smithi142.stderr:2024-05-08T01:14:15.918+0000 7fcd4d74b7c0 -1 osd.1 0 log_to_monitors true
2024-05-08T01:14:15.975 INFO:tasks.ceph.cluster2.osd.0.smithi142.stderr:2024-05-08T01:14:15.994+0000 7f7625dd97c0 -1 osd.0 0 log_to_monitors true
2024-05-08T01:14:15.989 INFO:tasks.ceph.cluster2.osd.2.smithi142.stderr:2024-05-08T01:14:16.006+0000 7f5007c817c0 -1 osd.2 0 log_to_monitors true
2024-05-08T01:14:18.502 DEBUG:teuthology.orchestra.run.smithi142:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json
2024-05-08T01:14:18.829 INFO:teuthology.misc.health.smithi142.stdout:
2024-05-08T01:14:18.829 INFO:teuthology.misc.health.smithi142.stdout:{"epoch":7,"fsid":"e7480526-62e8-4ed0-8efe-58a8e36c6942","created":"2024-05-08T01:14:10.569210+0000","modified":"2024-05-08T01:14:17.909787+0000","last_up_change":"0.000000","last_in_change":"2024-05-08T01:14:11.492301+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0b037d45-97f6-4d6b-9827-0fae9be5eede","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b14aaee3-26c1-43d8-bd53-f04a32fb66de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"308ad5c2-db06-434a-8986-73794c7f77e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
2024-05-08T01:14:18.847 DEBUG:teuthology.misc:0 of 3 OSDs are up
2024-05-08T01:14:19.704 INFO:tasks.ceph.cluster2.osd.2.smithi142.stderr:2024-05-08T01:14:19.722+0000 7f5003c2a640 -1 osd.2 0 waiting for initial osdmap
2024-05-08T01:14:19.735 INFO:tasks.ceph.cluster2.osd.2.smithi142.stderr:2024-05-08T01:14:19.754+0000 7f4ffba5f640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory
2024-05-08T01:14:19.753 INFO:tasks.ceph.cluster2.osd.0.smithi142.stderr:2024-05-08T01:14:19.770+0000 7f7621d82640 -1 osd.0 0 waiting for initial osdmap
2024-05-08T01:14:19.760 INFO:tasks.ceph.cluster2.osd.1.smithi142.stderr:2024-05-08T01:14:19.778+0000 7fcd49f06640 -1 osd.1 0 waiting for initial osdmap
2024-05-08T01:14:19.779 INFO:tasks.ceph.cluster2.osd.0.smithi142.stderr:2024-05-08T01:14:19.798+0000 7f7619bb7640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory
2024-05-08T01:14:19.792 INFO:tasks.ceph.cluster2.osd.1.smithi142.stderr:2024-05-08T01:14:19.810+0000 7fcd41529640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory
2024-05-08T01:14:25.150 DEBUG:teuthology.orchestra.run.smithi142:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json
2024-05-08T01:14:25.450 INFO:teuthology.misc.health.smithi142.stdout:
2024-05-08T01:14:25.450 INFO:teuthology.misc.health.smithi142.stdout:{"epoch":12,"fsid":"e7480526-62e8-4ed0-8efe-58a8e36c6942","created":"2024-05-08T01:14:10.569210+0000","modified":"2024-05-08T01:14:24.805982+0000","last_up_change":"2024-05-08T01:14:20.782949+0000","last_in_change":"2024-05-08T01:14:11.492301+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T01:14:21.961770+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b037d45-97f6-4d6b-9827-0fae9be5eede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":420380403},{"type":"v1","addr":"172.21.15.142:6807","nonce":420380403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":420380403},{"type":"v1","addr":"172.21.15.142:6813","nonce":420380403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6821","nonce":420380403},{"type":"v1","addr":"172.21.15.142:6823","nonce":420380403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":420380403},{"type":"v1","addr":"172.21.15.142:6819","nonce":420380403}]},"public_addr":"172.21.15.142:6807/420380403","cluster_addr":"172.21.15.142:6813/420380403","heartbeat_back_addr":"172.21.15.142:6823/420380403","heartbeat_front_addr":"172.21.15.142:6819/420380403","state":["exists","up"]},{"osd":1,"uuid":"b14aaee3-26c1-43d8-bd53-f04a32fb66de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":3687546466},{"type":"v1","addr":"172.21.15.142:6801","nonce":3687546466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":3687546466},{"type":"v1","addr":"172.21.15.142:6805","nonce":3687546466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":3687546466},{"type":"v1","addr":"172.21.15.142:6817","nonce":3687546466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":3687546466},{"type":"v1","addr":"172.21.15.142:6811","nonce":3687546466}]},"public_addr":"172.21.15.142:6801/3687546466","cluster_addr":"172.21.15.142:6805/3687546466","heartbeat_back_addr":"172.21.15.142:6817/3687546466","heartbeat_front_addr":"172.21.15.142:6811/3687546466","state":["exists","up"]},{"osd":2,"uuid":"308ad5c2-db06-434a-8986-73794c7f77e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6803","nonce":2528577693},{"type":"v1","addr":"172.21.15.142:6806","nonce":2528577693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6809","nonce":2528577693},{"type":"v1","addr":"172.21.15.142:6812","nonce":2528577693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2528577693},{"type":"v1","addr":"172.21.15.142:6822","nonce":2528577693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6815","nonce":2528577693},{"type":"v1","addr":"172.21.15.142:6818","nonce":2528577693}]},"public_addr":"172.21.15.142:6806/2528577693","cluster_addr":"172.21.15.142:6812/2528577693","heartbeat_back_addr":"172.21.15.142:6822/2528577693","heartbeat_front_addr":"172.21.15.142:6818/2528577693","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:14:16.954207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:14:16.922438+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:14:17.044924+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
2024-05-08T01:14:25.467 DEBUG:teuthology.misc:3 of 3 OSDs are up
2024-05-08T01:14:25.467 INFO:tasks.ceph:Creating RBD pool
2024-05-08T01:14:25.468 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster cluster2 osd pool create rbd 8
2024-05-08T01:14:25.792 INFO:teuthology.orchestra.run.smithi142.stderr:pool 'rbd' created
2024-05-08T01:14:25.814 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it
2024-05-08T01:14:26.794 INFO:teuthology.orchestra.run.smithi142.stderr:enabled application 'rbd' on pool 'rbd'
2024-05-08T01:14:26.825 INFO:tasks.ceph:Starting mds daemons in cluster cluster2...
2024-05-08T01:14:26.825 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean...
2024-05-08T01:14:26.825 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available
2024-05-08T01:14:26.826 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json
2024-05-08T01:14:26.826 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting
2024-05-08T01:14:27.156 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:14:27.173 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":2499830525},{"type":"v1","addr":"172.21.15.106:6827","nonce":2499830525}]},"active_addr":"172.21.15.106:6827/2499830525","active_change":"2024-05-08T01:14:13.898868+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace.  Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB.  Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":561500690}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":3203847255}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":3373565411}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":3309944374}]}]}
2024-05-08T01:14:27.175 INFO:tasks.ceph.ceph_manager.cluster2:mgr available!
2024-05-08T01:14:27.175 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up
2024-05-08T01:14:27.175 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json
2024-05-08T01:14:27.447 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:14:27.448 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":14,"fsid":"e7480526-62e8-4ed0-8efe-58a8e36c6942","created":"2024-05-08T01:14:10.569210+0000","modified":"2024-05-08T01:14:26.813747+0000","last_up_change":"2024-05-08T01:14:20.782949+0000","last_in_change":"2024-05-08T01:14:11.492301+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T01:14:21.961770+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-08T01:14:25.806066+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b037d45-97f6-4d6b-9827-0fae9be5eede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":420380403},{"type":"v1","addr":"172.21.15.142:6807","nonce":420380403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":420380403},{"type":"v1","addr":"172.21.15.142:6813","nonce":420380403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6821","nonce":420380403},{"type":"v1","addr":"172.21.15.142:6823","nonce":420380403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":420380403},{"type":"v1","addr":"172.21.15.142:6819","nonce":420380403}]},"public_addr":"172.21.15.142:6807/420380403","cluster_addr":"172.21.15.142:6813/420380403","heartbeat_back_addr":"172.21.15.142:6823/420380403","heartbeat_front_addr":"172.21.15.142:6819/420380403","state":["exists","up"]},{"osd":1,"uuid":"b14aaee3-26c1-43d8-bd53-f04a32fb66de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":3687546466},{"type":"v1","addr":"172.21.15.142:6801","nonce":3687546466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":3687546466},{"type":"v1","addr":"172.21.15.142:6805","nonce":3687546466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":3687546466},{"type":"v1","addr":"172.21.15.142:6817","nonce":3687546466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":3687546466},{"type":"v1","addr":"172.21.15.142:6811","nonce":3687546466}]},"public_addr":"172.21.15.142:6801/3687546466","cluster_addr":"172.21.15.142:6805/3687546466","heartbeat_back_addr":"172.21.15.142:6817/3687546466","heartbeat_front_addr":"172.21.15.142:6811/3687546466","state":["exists","up"]},{"osd":2,"uuid":"308ad5c2-db06-434a-8986-73794c7f77e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6803","nonce":2528577693},{"type":"v1","addr":"172.21.15.142:6806","nonce":2528577693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6809","nonce":2528577693},{"type":"v1","addr":"172.21.15.142:6812","nonce":2528577693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2528577693},{"type":"v1","addr":"172.21.15.142:6822","nonce":2528577693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6815","nonce":2528577693},{"type":"v1","addr":"172.21.15.142:6818","nonce":2528577693}]},"public_addr":"172.21.15.142:6806/2528577693","cluster_addr":"172.21.15.142:6812/2528577693","heartbeat_back_addr":"172.21.15.142:6822/2528577693","heartbeat_front_addr":"172.21.15.142:6818/2528577693","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:14:16.954207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:14:16.922438+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:14:17.044924+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
2024-05-08T01:14:27.465 INFO:tasks.ceph.ceph_manager.cluster2:all up!
2024-05-08T01:14:27.465 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json
2024-05-08T01:14:27.760 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:14:27.760 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":14,"fsid":"e7480526-62e8-4ed0-8efe-58a8e36c6942","created":"2024-05-08T01:14:10.569210+0000","modified":"2024-05-08T01:14:26.813747+0000","last_up_change":"2024-05-08T01:14:20.782949+0000","last_in_change":"2024-05-08T01:14:11.492301+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T01:14:21.961770+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-08T01:14:25.806066+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b037d45-97f6-4d6b-9827-0fae9be5eede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":420380403},{"type":"v1","addr":"172.21.15.142:6807","nonce":420380403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":420380403},{"type":"v1","addr":"172.21.15.142:6813","nonce":420380403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6821","nonce":420380403},{"type":"v1","addr":"172.21.15.142:6823","nonce":420380403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":420380403},{"type":"v1","addr":"172.21.15.142:6819","nonce":420380403}]},"public_addr":"172.21.15.142:6807/420380403","cluster_addr":"172.21.15.142:6813/420380403","heartbeat_back_addr":"172.21.15.142:6823/420380403","heartbeat_front_addr":"172.21.15.142:6819/420380403","state":["exists","up"]},{"osd":1,"uuid":"b14aaee3-26c1-43d8-bd53-f04a32fb66de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":3687546466},{"type":"v1","addr":"172.21.15.142:6801","nonce":3687546466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":3687546466},{"type":"v1","addr":"172.21.15.142:6805","nonce":3687546466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":3687546466},{"type":"v1","addr":"172.21.15.142:6817","nonce":3687546466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":3687546466},{"type":"v1","addr":"172.21.15.142:6811","nonce":3687546466}]},"public_addr":"172.21.15.142:6801/3687546466","cluster_addr":"172.21.15.142:6805/3687546466","heartbeat_back_addr":"172.21.15.142:6817/3687546466","heartbeat_front_addr":"172.21.15.142:6811/3687546466","state":["exists","up"]},{"osd":2,"uuid":"308ad5c2-db06-434a-8986-73794c7f77e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6803","nonce":2528577693},{"type":"v1","addr":"172.21.15.142:6806","nonce":2528577693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6809","nonce":2528577693},{"type":"v1","addr":"172.21.15.142:6812","nonce":2528577693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2528577693},{"type":"v1","addr":"172.21.15.142:6822","nonce":2528577693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6815","nonce":2528577693},{"type":"v1","addr":"172.21.15.142:6818","nonce":2528577693}]},"public_addr":"172.21.15.142:6806/2528577693","cluster_addr":"172.21.15.142:6812/2528577693","heartbeat_back_addr":"172.21.15.142:6822/2528577693","heartbeat_front_addr":"172.21.15.142:6818/2528577693","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:14:16.954207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:14:16.922438+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:14:17.044924+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
2024-05-08T01:14:27.779 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats
2024-05-08T01:14:27.779 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats
2024-05-08T01:14:27.780 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats
2024-05-08T01:14:27.934 INFO:teuthology.orchestra.run.smithi142.stdout:34359738374
2024-05-08T01:14:27.935 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1
2024-05-08T01:14:27.937 INFO:teuthology.orchestra.run.smithi142.stdout:34359738374
2024-05-08T01:14:27.938 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0
2024-05-08T01:14:27.943 INFO:teuthology.orchestra.run.smithi142.stdout:34359738374
2024-05-08T01:14:27.943 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2
2024-05-08T01:14:28.222 INFO:teuthology.orchestra.run.smithi142.stdout:34359738372
2024-05-08T01:14:28.239 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.1
2024-05-08T01:14:28.262 INFO:teuthology.orchestra.run.smithi142.stdout:34359738372
2024-05-08T01:14:28.270 INFO:teuthology.orchestra.run.smithi142.stdout:34359738372
2024-05-08T01:14:28.281 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.0
2024-05-08T01:14:28.289 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.2
2024-05-08T01:14:29.239 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1
2024-05-08T01:14:29.282 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0
2024-05-08T01:14:29.289 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2
2024-05-08T01:14:29.546 INFO:teuthology.orchestra.run.smithi142.stdout:34359738373
2024-05-08T01:14:29.563 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1
2024-05-08T01:14:29.593 INFO:teuthology.orchestra.run.smithi142.stdout:34359738373
2024-05-08T01:14:29.611 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0
2024-05-08T01:14:29.632 INFO:teuthology.orchestra.run.smithi142.stdout:34359738373
2024-05-08T01:14:29.649 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2
2024-05-08T01:14:30.564 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1
2024-05-08T01:14:30.612 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0
2024-05-08T01:14:30.650 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2
2024-05-08T01:14:30.902 INFO:teuthology.orchestra.run.smithi142.stdout:34359738374
2024-05-08T01:14:30.909 INFO:teuthology.orchestra.run.smithi142.stdout:34359738374
2024-05-08T01:14:30.921 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.0
2024-05-08T01:14:30.921 DEBUG:teuthology.parallel:result is None
2024-05-08T01:14:30.928 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.1
2024-05-08T01:14:30.928 DEBUG:teuthology.parallel:result is None
2024-05-08T01:14:30.981 INFO:teuthology.orchestra.run.smithi142.stdout:34359738374
2024-05-08T01:14:30.998 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.2
2024-05-08T01:14:30.998 DEBUG:teuthology.parallel:result is None
2024-05-08T01:14:30.998 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean
2024-05-08T01:14:30.998 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json
2024-05-08T01:14:31.279 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:14:31.279 INFO:teuthology.orchestra.run.smithi142.stderr:dumped all
2024-05-08T01:14:31.296 INFO:teuthology.orchestra.run.smithi142.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-05-08T01:14:29.909314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82976,"kb_used_data":1520,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032544,"statfs":{"total":289910292480,"available":289825325056,"internally_reserved":0,"allocated":1556480,"data_stored":1282004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.091834"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T01:14:26.833531+0000","last_change":"2024-05-08T01:14:26.833531+0000","last_active":"2024-05-08T01:14:26.833531+0000","last_peered":"2024-05-08T01:14:26.833531+0000","last_clean":"2024-05-08T01:14:26.833531+0000","last_became_active":"2024-05-08T01:14:26.832964+0000","last_became_peered":"2024-05-08T01:14:26.832964+0000","last_unstale":"2024-05-08T01:14:26.833531+0000","last_undegraded":"2024-05-08T01:14:26.833531+0000","last_fullsized":"2024-05-08T01:14:26.833531+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:20:04.101911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T01:14:26.833601+0000","last_change":"2024-05-08T01:14:26.833601+0000","last_active":"2024-05-08T01:14:26.833601+0000","last_peered":"2024-05-08T01:14:26.833601+0000","last_clean":"2024-05-08T01:14:26.833601+0000","last_became_active":"2024-05-08T01:14:26.833073+0000","last_became_peered":"2024-05-08T01:14:26.833073+0000","last_unstale":"2024-05-08T01:14:26.833601+0000","last_undegraded":"2024-05-08T01:14:26.833601+0000","last_fullsized":"2024-05-08T01:14:26.833601+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:53:33.175384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T01:14:26.833188+0000","last_change":"2024-05-08T01:14:26.833188+0000","last_active":"2024-05-08T01:14:26.833188+0000","last_peered":"2024-05-08T01:14:26.833188+0000","last_clean":"2024-05-08T01:14:26.833188+0000","last_became_active":"2024-05-08T01:14:26.832760+0000","last_became_peered":"2024-05-08T01:14:26.832760+0000","last_unstale":"2024-05-08T01:14:26.833188+0000","last_undegraded":"2024-05-08T01:14:26.833188+0000","last_fullsized":"2024-05-08T01:14:26.833188+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:18:07.784432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T01:14:26.833124+0000","last_change":"2024-05-08T01:14:26.833124+0000","last_active":"2024-05-08T01:14:26.833124+0000","last_peered":"2024-05-08T01:14:26.833124+0000","last_clean":"2024-05-08T01:14:26.833124+0000","last_became_active":"2024-05-08T01:14:26.832756+0000","last_became_peered":"2024-05-08T01:14:26.832756+0000","last_unstale":"2024-05-08T01:14:26.833124+0000","last_undegraded":"2024-05-08T01:14:26.833124+0000","last_fullsized":"2024-05-08T01:14:26.833124+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T09:28:15.464050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-08T01:14:27.832719+0000","last_change":"2024-05-08T01:14:26.833102+0000","last_active":"2024-05-08T01:14:27.832719+0000","last_peered":"2024-05-08T01:14:27.832719+0000","last_clean":"2024-05-08T01:14:27.832719+0000","last_became_active":"2024-05-08T01:14:26.832771+0000","last_became_peered":"2024-05-08T01:14:26.832771+0000","last_unstale":"2024-05-08T01:14:27.832719+0000","last_undegraded":"2024-05-08T01:14:27.832719+0000","last_fullsized":"2024-05-08T01:14:27.832719+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:44:17.038811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T01:14:26.829716+0000","last_change":"2024-05-08T01:14:26.829716+0000","last_active":"2024-05-08T01:14:26.829716+0000","last_peered":"2024-05-08T01:14:26.829716+0000","last_clean":"2024-05-08T01:14:26.829716+0000","last_became_active":"2024-05-08T01:14:26.829273+0000","last_became_peered":"2024-05-08T01:14:26.829273+0000","last_unstale":"2024-05-08T01:14:26.829716+0000","last_undegraded":"2024-05-08T01:14:26.829716+0000","last_fullsized":"2024-05-08T01:14:26.829716+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T09:21:37.969083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T01:14:26.829079+0000","last_change":"2024-05-08T01:14:26.829079+0000","last_active":"2024-05-08T01:14:26.829079+0000","last_peered":"2024-05-08T01:14:26.829079+0000","last_clean":"2024-05-08T01:14:26.829079+0000","last_became_active":"2024-05-08T01:14:26.828803+0000","last_became_peered":"2024-05-08T01:14:26.828803+0000","last_unstale":"2024-05-08T01:14:26.829079+0000","last_undegraded":"2024-05-08T01:14:26.829079+0000","last_fullsized":"2024-05-08T01:14:26.829079+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:53:42.887219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T01:14:26.828790+0000","last_change":"2024-05-08T01:14:26.828790+0000","last_active":"2024-05-08T01:14:26.828790+0000","last_peered":"2024-05-08T01:14:26.828790+0000","last_clean":"2024-05-08T01:14:26.828790+0000","last_became_active":"2024-05-08T01:14:26.828455+0000","last_became_peered":"2024-05-08T01:14:26.828455+0000","last_unstale":"2024-05-08T01:14:26.828790+0000","last_undegraded":"2024-05-08T01:14:26.828790+0000","last_fullsized":"2024-05-08T01:14:26.828790+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:44:27.557965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":143,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T01:14:26.825131+0000","last_change":"2024-05-08T01:14:24.071859+0000","last_active":"2024-05-08T01:14:26.825131+0000","last_peered":"2024-05-08T01:14:26.825131+0000","last_clean":"2024-05-08T01:14:26.825131+0000","last_became_active":"2024-05-08T01:14:24.069533+0000","last_became_peered":"2024-05-08T01:14:24.069533+0000","last_unstale":"2024-05-08T01:14:26.825131+0000","last_undegraded":"2024-05-08T01:14:26.825131+0000","last_fullsized":"2024-05-08T01:14:26.825131+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:22.792920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:22.792920+0000","last_clean_scrub_stamp":"2024-05-08T01:14:22.792920+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:27:03.487086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":33756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":716800,"data_stored":624124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":716800,"data_stored":624124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}}
2024-05-08T01:14:31.297 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json
2024-05-08T01:14:31.564 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:14:31.564 INFO:teuthology.orchestra.run.smithi142.stderr:dumped all
2024-05-08T01:14:31.581 INFO:teuthology.orchestra.run.smithi142.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-05-08T01:14:29.909314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82976,"kb_used_data":1520,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032544,"statfs":{"total":289910292480,"available":289825325056,"internally_reserved":0,"allocated":1556480,"data_stored":1282004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.091834"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T01:14:26.833531+0000","last_change":"2024-05-08T01:14:26.833531+0000","last_active":"2024-05-08T01:14:26.833531+0000","last_peered":"2024-05-08T01:14:26.833531+0000","last_clean":"2024-05-08T01:14:26.833531+0000","last_became_active":"2024-05-08T01:14:26.832964+0000","last_became_peered":"2024-05-08T01:14:26.832964+0000","last_unstale":"2024-05-08T01:14:26.833531+0000","last_undegraded":"2024-05-08T01:14:26.833531+0000","last_fullsized":"2024-05-08T01:14:26.833531+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:20:04.101911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T01:14:26.833601+0000","last_change":"2024-05-08T01:14:26.833601+0000","last_active":"2024-05-08T01:14:26.833601+0000","last_peered":"2024-05-08T01:14:26.833601+0000","last_clean":"2024-05-08T01:14:26.833601+0000","last_became_active":"2024-05-08T01:14:26.833073+0000","last_became_peered":"2024-05-08T01:14:26.833073+0000","last_unstale":"2024-05-08T01:14:26.833601+0000","last_undegraded":"2024-05-08T01:14:26.833601+0000","last_fullsized":"2024-05-08T01:14:26.833601+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:53:33.175384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T01:14:26.833188+0000","last_change":"2024-05-08T01:14:26.833188+0000","last_active":"2024-05-08T01:14:26.833188+0000","last_peered":"2024-05-08T01:14:26.833188+0000","last_clean":"2024-05-08T01:14:26.833188+0000","last_became_active":"2024-05-08T01:14:26.832760+0000","last_became_peered":"2024-05-08T01:14:26.832760+0000","last_unstale":"2024-05-08T01:14:26.833188+0000","last_undegraded":"2024-05-08T01:14:26.833188+0000","last_fullsized":"2024-05-08T01:14:26.833188+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:18:07.784432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T01:14:26.833124+0000","last_change":"2024-05-08T01:14:26.833124+0000","last_active":"2024-05-08T01:14:26.833124+0000","last_peered":"2024-05-08T01:14:26.833124+0000","last_clean":"2024-05-08T01:14:26.833124+0000","last_became_active":"2024-05-08T01:14:26.832756+0000","last_became_peered":"2024-05-08T01:14:26.832756+0000","last_unstale":"2024-05-08T01:14:26.833124+0000","last_undegraded":"2024-05-08T01:14:26.833124+0000","last_fullsized":"2024-05-08T01:14:26.833124+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T09:28:15.464050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-08T01:14:27.832719+0000","last_change":"2024-05-08T01:14:26.833102+0000","last_active":"2024-05-08T01:14:27.832719+0000","last_peered":"2024-05-08T01:14:27.832719+0000","last_clean":"2024-05-08T01:14:27.832719+0000","last_became_active":"2024-05-08T01:14:26.832771+0000","last_became_peered":"2024-05-08T01:14:26.832771+0000","last_unstale":"2024-05-08T01:14:27.832719+0000","last_undegraded":"2024-05-08T01:14:27.832719+0000","last_fullsized":"2024-05-08T01:14:27.832719+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:44:17.038811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T01:14:26.829716+0000","last_change":"2024-05-08T01:14:26.829716+0000","last_active":"2024-05-08T01:14:26.829716+0000","last_peered":"2024-05-08T01:14:26.829716+0000","last_clean":"2024-05-08T01:14:26.829716+0000","last_became_active":"2024-05-08T01:14:26.829273+0000","last_became_peered":"2024-05-08T01:14:26.829273+0000","last_unstale":"2024-05-08T01:14:26.829716+0000","last_undegraded":"2024-05-08T01:14:26.829716+0000","last_fullsized":"2024-05-08T01:14:26.829716+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T09:21:37.969083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T01:14:26.829079+0000","last_change":"2024-05-08T01:14:26.829079+0000","last_active":"2024-05-08T01:14:26.829079+0000","last_peered":"2024-05-08T01:14:26.829079+0000","last_clean":"2024-05-08T01:14:26.829079+0000","last_became_active":"2024-05-08T01:14:26.828803+0000","last_became_peered":"2024-05-08T01:14:26.828803+0000","last_unstale":"2024-05-08T01:14:26.829079+0000","last_undegraded":"2024-05-08T01:14:26.829079+0000","last_fullsized":"2024-05-08T01:14:26.829079+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:53:42.887219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T01:14:26.828790+0000","last_change":"2024-05-08T01:14:26.828790+0000","last_active":"2024-05-08T01:14:26.828790+0000","last_peered":"2024-05-08T01:14:26.828790+0000","last_clean":"2024-05-08T01:14:26.828790+0000","last_became_active":"2024-05-08T01:14:26.828455+0000","last_became_peered":"2024-05-08T01:14:26.828455+0000","last_unstale":"2024-05-08T01:14:26.828790+0000","last_undegraded":"2024-05-08T01:14:26.828790+0000","last_fullsized":"2024-05-08T01:14:26.828790+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:44:27.557965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":143,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T01:14:26.825131+0000","last_change":"2024-05-08T01:14:24.071859+0000","last_active":"2024-05-08T01:14:26.825131+0000","last_peered":"2024-05-08T01:14:26.825131+0000","last_clean":"2024-05-08T01:14:26.825131+0000","last_became_active":"2024-05-08T01:14:24.069533+0000","last_became_peered":"2024-05-08T01:14:24.069533+0000","last_unstale":"2024-05-08T01:14:26.825131+0000","last_undegraded":"2024-05-08T01:14:26.825131+0000","last_fullsized":"2024-05-08T01:14:26.825131+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:22.792920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:22.792920+0000","last_clean_scrub_stamp":"2024-05-08T01:14:22.792920+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:27:03.487086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":33756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":716800,"data_stored":624124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":716800,"data_stored":624124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}}
2024-05-08T01:14:31.581 INFO:tasks.ceph.ceph_manager.cluster2:clean!
2024-05-08T01:14:31.582 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy...
2024-05-08T01:14:31.582 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy
2024-05-08T01:14:31.582 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json
2024-05-08T01:14:31.884 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:14:31.884 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]}
2024-05-08T01:14:31.901 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done
2024-05-08T01:14:31.902 INFO:teuthology.run_tasks:Running task exec...
2024-05-08T01:14:31.911 INFO:teuthology.task.exec:Executing custom commands...
2024-05-08T01:14:31.911 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi142.front.sepia.ceph.com
2024-05-08T01:14:31.911 DEBUG:teuthology.orchestra.run.smithi142:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"''
2024-05-08T01:14:32.265 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.4]
2024-05-08T01:14:32.265 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQCw0TpmqGRXIRAAlU/iQY4uUexE/s/6Np/5+w==
2024-05-08T01:14:32.265 INFO:teuthology.orchestra.run.smithi142.stdout:	caps mon = "profile rbd-mirror"
2024-05-08T01:14:32.265 INFO:teuthology.orchestra.run.smithi142.stdout:	caps osd = "profile rbd"
2024-05-08T01:14:32.265 INFO:teuthology.orchestra.run.smithi142.stderr:updated caps for client.mirror.4
2024-05-08T01:14:32.283 DEBUG:teuthology.orchestra.run.smithi142:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"''
2024-05-08T01:14:32.645 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.5]
2024-05-08T01:14:32.645 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQCw0Tpmj0c/JhAAhBhBGcmNBHfPQbMZ+etjwg==
2024-05-08T01:14:32.645 INFO:teuthology.orchestra.run.smithi142.stdout:	caps mon = "profile rbd-mirror"
2024-05-08T01:14:32.645 INFO:teuthology.orchestra.run.smithi142.stdout:	caps osd = "profile rbd"
2024-05-08T01:14:32.646 INFO:teuthology.orchestra.run.smithi142.stderr:updated caps for client.mirror.5
2024-05-08T01:14:32.663 DEBUG:teuthology.orchestra.run.smithi142:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"''
2024-05-08T01:14:33.030 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.6]
2024-05-08T01:14:33.031 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQCw0TpmGdLqKhAAwWDHt75drquNIqZdMDBOHg==
2024-05-08T01:14:33.031 INFO:teuthology.orchestra.run.smithi142.stdout:	caps mon = "profile rbd-mirror"
2024-05-08T01:14:33.031 INFO:teuthology.orchestra.run.smithi142.stdout:	caps osd = "profile rbd"
2024-05-08T01:14:33.031 INFO:teuthology.orchestra.run.smithi142.stderr:updated caps for client.mirror.6
2024-05-08T01:14:33.048 DEBUG:teuthology.orchestra.run.smithi142:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"''
2024-05-08T01:14:33.390 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.4]
2024-05-08T01:14:33.390 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDU0Tpm4uyjMRAAcxjG1afFqDAjGw/SeACdDQ==
2024-05-08T01:14:33.390 INFO:teuthology.orchestra.run.smithi142.stdout:	caps mon = "profile rbd-mirror"
2024-05-08T01:14:33.391 INFO:teuthology.orchestra.run.smithi142.stdout:	caps osd = "profile rbd"
2024-05-08T01:14:33.391 INFO:teuthology.orchestra.run.smithi142.stderr:updated caps for client.mirror.4
2024-05-08T01:14:33.407 DEBUG:teuthology.orchestra.run.smithi142:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"''
2024-05-08T01:14:33.750 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.5]
2024-05-08T01:14:33.751 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDU0TpmGYIeNxAAZ2F9iLCpLGyGqyuT6HE3lg==
2024-05-08T01:14:33.751 INFO:teuthology.orchestra.run.smithi142.stdout:	caps mon = "profile rbd-mirror"
2024-05-08T01:14:33.751 INFO:teuthology.orchestra.run.smithi142.stdout:	caps osd = "profile rbd"
2024-05-08T01:14:33.751 INFO:teuthology.orchestra.run.smithi142.stderr:updated caps for client.mirror.5
2024-05-08T01:14:33.769 DEBUG:teuthology.orchestra.run.smithi142:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"''
2024-05-08T01:14:34.114 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.6]
2024-05-08T01:14:34.114 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDV0TpmzzlzABAAI23N3VUmx57gjGU5fk5zsg==
2024-05-08T01:14:34.114 INFO:teuthology.orchestra.run.smithi142.stdout:	caps mon = "profile rbd-mirror"
2024-05-08T01:14:34.114 INFO:teuthology.orchestra.run.smithi142.stdout:	caps osd = "profile rbd"
2024-05-08T01:14:34.114 INFO:teuthology.orchestra.run.smithi142.stderr:updated caps for client.mirror.6
2024-05-08T01:14:34.131 INFO:teuthology.run_tasks:Running task exec...
2024-05-08T01:14:34.144 INFO:teuthology.task.exec:Executing custom commands...
2024-05-08T01:14:34.144 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi142.front.sepia.ceph.com
2024-05-08T01:14:34.144 DEBUG:teuthology.orchestra.run.smithi142:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"''
2024-05-08T01:14:34.513 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror]
2024-05-08T01:14:34.513 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQCw0TpmuEY9BxAA8mF/n1ILdufosemF9zXDqQ==
2024-05-08T01:14:34.514 INFO:teuthology.orchestra.run.smithi142.stdout:	caps mon = "profile rbd-mirror-peer"
2024-05-08T01:14:34.514 INFO:teuthology.orchestra.run.smithi142.stdout:	caps osd = "profile rbd"
2024-05-08T01:14:34.514 INFO:teuthology.orchestra.run.smithi142.stderr:updated caps for client.mirror
2024-05-08T01:14:34.533 DEBUG:teuthology.orchestra.run.smithi142:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"''
2024-05-08T01:14:34.912 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.0]
2024-05-08T01:14:34.912 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQCw0TpmKDRoDRAAkLFmFOTGyffdUBRbQo1QZQ==
2024-05-08T01:14:34.912 INFO:teuthology.orchestra.run.smithi142.stdout:	caps mon = "profile rbd-mirror"
2024-05-08T01:14:34.912 INFO:teuthology.orchestra.run.smithi142.stdout:	caps osd = "profile rbd"
2024-05-08T01:14:34.912 INFO:teuthology.orchestra.run.smithi142.stderr:updated caps for client.mirror.0
2024-05-08T01:14:34.930 DEBUG:teuthology.orchestra.run.smithi142:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"''
2024-05-08T01:14:35.282 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.1]
2024-05-08T01:14:35.282 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQCw0TpmpBpHEhAAjBfzt3Kt3/GtZFeRBPopvA==
2024-05-08T01:14:35.282 INFO:teuthology.orchestra.run.smithi142.stdout:	caps mon = "profile rbd-mirror"
2024-05-08T01:14:35.283 INFO:teuthology.orchestra.run.smithi142.stdout:	caps osd = "profile rbd"
2024-05-08T01:14:35.283 INFO:teuthology.orchestra.run.smithi142.stderr:updated caps for client.mirror.1
2024-05-08T01:14:35.300 DEBUG:teuthology.orchestra.run.smithi142:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"''
2024-05-08T01:14:35.653 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.2]
2024-05-08T01:14:35.653 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQCw0TpmRgxKFxAA6BY1rU8lpefCMcZbfYMIUw==
2024-05-08T01:14:35.653 INFO:teuthology.orchestra.run.smithi142.stdout:	caps mon = "profile rbd-mirror"
2024-05-08T01:14:35.653 INFO:teuthology.orchestra.run.smithi142.stdout:	caps osd = "profile rbd"
2024-05-08T01:14:35.654 INFO:teuthology.orchestra.run.smithi142.stderr:updated caps for client.mirror.2
2024-05-08T01:14:35.670 DEBUG:teuthology.orchestra.run.smithi142:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"''
2024-05-08T01:14:36.042 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.3]
2024-05-08T01:14:36.045 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQCw0TpmpOVZHBAAhgCKU6ZG61kmmQ+H9ubnLQ==
2024-05-08T01:14:36.045 INFO:teuthology.orchestra.run.smithi142.stdout:	caps mon = "profile rbd-mirror"
2024-05-08T01:14:36.045 INFO:teuthology.orchestra.run.smithi142.stdout:	caps osd = "profile rbd"
2024-05-08T01:14:36.046 INFO:teuthology.orchestra.run.smithi142.stderr:updated caps for client.mirror.3
2024-05-08T01:14:36.062 DEBUG:teuthology.orchestra.run.smithi142:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"''
2024-05-08T01:14:36.397 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror]
2024-05-08T01:14:36.398 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDU0Tpm6jvlFxAAG4eHE76fsVI+LBp4dAWL/w==
2024-05-08T01:14:36.398 INFO:teuthology.orchestra.run.smithi142.stdout:	caps mon = "profile rbd-mirror-peer"
2024-05-08T01:14:36.398 INFO:teuthology.orchestra.run.smithi142.stdout:	caps osd = "profile rbd"
2024-05-08T01:14:36.398 INFO:teuthology.orchestra.run.smithi142.stderr:updated caps for client.mirror
2024-05-08T01:14:36.415 DEBUG:teuthology.orchestra.run.smithi142:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"''
2024-05-08T01:14:36.748 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.0]
2024-05-08T01:14:36.748 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDU0TpmPyrvHBAAw8aH7W5vIanto3wgH7uMkA==
2024-05-08T01:14:36.748 INFO:teuthology.orchestra.run.smithi142.stdout:	caps mon = "profile rbd-mirror"
2024-05-08T01:14:36.748 INFO:teuthology.orchestra.run.smithi142.stdout:	caps osd = "profile rbd"
2024-05-08T01:14:36.748 INFO:teuthology.orchestra.run.smithi142.stderr:updated caps for client.mirror.0
2024-05-08T01:14:36.766 DEBUG:teuthology.orchestra.run.smithi142:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"''
2024-05-08T01:14:37.113 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.1]
2024-05-08T01:14:37.113 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDU0TpmB3RSIhAAAG/lLDbxOoO6j3AemEUzjA==
2024-05-08T01:14:37.113 INFO:teuthology.orchestra.run.smithi142.stdout:	caps mon = "profile rbd-mirror"
2024-05-08T01:14:37.113 INFO:teuthology.orchestra.run.smithi142.stdout:	caps osd = "profile rbd"
2024-05-08T01:14:37.114 INFO:teuthology.orchestra.run.smithi142.stderr:updated caps for client.mirror.1
2024-05-08T01:14:37.130 DEBUG:teuthology.orchestra.run.smithi142:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"''
2024-05-08T01:14:37.463 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.2]
2024-05-08T01:14:37.464 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDU0TpmFPtZJxAAOQPBuOoufjGuEmNq37QuOg==
2024-05-08T01:14:37.464 INFO:teuthology.orchestra.run.smithi142.stdout:	caps mon = "profile rbd-mirror"
2024-05-08T01:14:37.464 INFO:teuthology.orchestra.run.smithi142.stdout:	caps osd = "profile rbd"
2024-05-08T01:14:37.464 INFO:teuthology.orchestra.run.smithi142.stderr:updated caps for client.mirror.2
2024-05-08T01:14:37.483 DEBUG:teuthology.orchestra.run.smithi142:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"''
2024-05-08T01:14:37.819 INFO:teuthology.orchestra.run.smithi142.stdout:[client.mirror.3]
2024-05-08T01:14:37.819 INFO:teuthology.orchestra.run.smithi142.stdout:	key = AQDU0TpmdeiXLBAAZ28sZWUQCPZ3+cDUY82DrA==
2024-05-08T01:14:37.819 INFO:teuthology.orchestra.run.smithi142.stdout:	caps mon = "profile rbd-mirror"
2024-05-08T01:14:37.820 INFO:teuthology.orchestra.run.smithi142.stdout:	caps osd = "profile rbd"
2024-05-08T01:14:37.820 INFO:teuthology.orchestra.run.smithi142.stderr:updated caps for client.mirror.3
2024-05-08T01:14:37.837 INFO:teuthology.run_tasks:Running task workunit...
2024-05-08T01:14:37.850 INFO:tasks.workunit:Pulling workunits from ref 9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2
2024-05-08T01:14:37.850 INFO:tasks.workunit:Making a separate scratch dir for every client...
2024-05-08T01:14:37.851 DEBUG:teuthology.orchestra.run.smithi142:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror
2024-05-08T01:14:37.855 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T01:14:37.856 INFO:teuthology.orchestra.run.smithi142.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory
2024-05-08T01:14:37.856 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror
2024-05-08T01:14:37.903 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror
2024-05-08T01:14:37.903 DEBUG:teuthology.orchestra.run.smithi142:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror
2024-05-08T01:14:37.955 INFO:tasks.workunit:timeout=3h
2024-05-08T01:14:37.955 INFO:tasks.workunit:cleanup=True
2024-05-08T01:14:37.955 DEBUG:teuthology.orchestra.run.smithi142:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2
2024-05-08T01:14:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'...
2024-05-08T01:15:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Note: switching to '9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2'.
2024-05-08T01:15:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
2024-05-08T01:15:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:You are in 'detached HEAD' state. You can look around, make experimental
2024-05-08T01:15:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:changes and commit them, and you can discard any commits you make in this
2024-05-08T01:15:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:state without impacting any branches by switching back to a branch.
2024-05-08T01:15:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
2024-05-08T01:15:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:If you want to create a new branch to retain commits you create, you may
2024-05-08T01:15:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:do so (now or later) by using -c with the switch command. Example:
2024-05-08T01:15:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
2024-05-08T01:15:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  git switch -c <new-branch-name>
2024-05-08T01:15:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
2024-05-08T01:15:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Or undo this operation with:
2024-05-08T01:15:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
2024-05-08T01:15:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  git switch -
2024-05-08T01:15:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
2024-05-08T01:15:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Turn off this advice by setting config variable advice.detachedHead to false
2024-05-08T01:15:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
2024-05-08T01:15:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:HEAD is now at 9aa7e81b5f5 Merge branch 'wip-65624-squid' of https://github.com/idryomov/ceph into HEAD
2024-05-08T01:15:32.154 DEBUG:teuthology.orchestra.run.smithi142:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror
2024-05-08T01:15:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done
2024-05-08T01:15:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io'
2024-05-08T01:15:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test
2024-05-08T01:15:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io
2024-05-08T01:15:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read
2024-05-08T01:15:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io'
2024-05-08T01:15:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs'
2024-05-08T01:15:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc
2024-05-08T01:15:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs'
2024-05-08T01:15:33.114 DEBUG:teuthology.orchestra.run.smithi142:> set -ex
2024-05-08T01:15:33.114 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout
2024-05-08T01:15:33.160 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror.sh on cluster1.client.mirror...
2024-05-08T01:15:33.163 INFO:tasks.workunit:Running workunit rbd/rbd_mirror.sh...
2024-05-08T01:15:33.163 DEBUG:teuthology.orchestra.run.smithi142:workunit test rbd/rbd_mirror.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=journal RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror.sh
2024-05-08T01:15:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -gt 0 ']'
2024-05-08T01:15:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal '!=' snapshot ']'
2024-05-08T01:15:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal '!=' journal ']'
2024-05-08T01:15:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror.sh
2024-05-08T01:15:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh
2024-05-08T01:15:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ type xmlstarlet
2024-05-08T01:15:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ XMLSTARLET=xmlstarlet
2024-05-08T01:15:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ RBD_MIRROR_INSTANCES=4
2024-05-08T01:15:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CLUSTER1=cluster1
2024-05-08T01:15:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CLUSTER2=cluster2
2024-05-08T01:15:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ PEER_CLUSTER_SUFFIX=
2024-05-08T01:15:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ POOL=mirror
2024-05-08T01:15:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ PARENT_POOL=mirror_parent
2024-05-08T01:15:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ NS1=ns1
2024-05-08T01:15:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ NS2=ns2
2024-05-08T01:15:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ TEMPDIR=
2024-05-08T01:15:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ID=mirror
2024-05-08T01:15:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,journaling
2024-05-08T01:15:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ MIRROR_USER_ID_PREFIX=mirror.
2024-05-08T01:15:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ RBD_MIRROR_MODE=journal
2024-05-08T01:15:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ MIRROR_POOL_MODE=pool
2024-05-08T01:15:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' journal = snapshot ']'
2024-05-08T01:15:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ export 'CEPH_ARGS=--id mirror'
2024-05-08T01:15:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS='--id mirror'
2024-05-08T01:15:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ LAST_MIRROR_INSTANCE=3
2024-05-08T01:15:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror.sh
2024-05-08T01:15:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src
2024-05-08T01:15:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src
2024-05-08T01:15:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_BIN=.
2024-05-08T01:15:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_SRC=.
2024-05-08T01:15:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' -e CMakeCache.txt ']'
2024-05-08T01:15:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 -gt 0 ']'
2024-05-08T01:15:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ setup
2024-05-08T01:15:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local c
2024-05-08T01:15:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ trap 'cleanup $?' INT TERM EXIT
2024-05-08T01:15:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ setup_tempdir
2024-05-08T01:15:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:15:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mktemp -d
2024-05-08T01:15:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ TEMPDIR=/tmp/tmp.3xddryFtVF
2024-05-08T01:15:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -z 1 ']'
2024-05-08T01:15:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ setup_pools cluster1 cluster2
2024-05-08T01:15:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:15:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local remote_cluster=cluster2
2024-05-08T01:15:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mon_map_file
2024-05-08T01:15:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mon_addr
2024-05-08T01:15:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local admin_key_file
2024-05-08T01:15:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local uuid
2024-05-08T01:15:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:15:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64
2024-05-08T01:15:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:pool 'mirror' already exists
2024-05-08T01:15:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:15:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64
2024-05-08T01:15:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:pool 'mirror_parent' already exists
2024-05-08T01:15:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:15:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 pool init mirror
2024-05-08T01:15:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:15:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 pool init mirror_parent
2024-05-08T01:15:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:15:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:15:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror pool
2024-05-08T01:15:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image
2024-05-08T01:15:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1
2024-05-08T01:15:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2
2024-05-08T01:15:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 pool
2024-05-08T01:15:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image
2024-05-08T01:15:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -z ']'
2024-05-08T01:15:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -z ']'
2024-05-08T01:15:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ peer_add cluster1 mirror cluster2
2024-05-08T01:15:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:15:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local client_cluster=cluster2
2024-05-08T01:15:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local remote_cluster=cluster2
2024-05-08T01:15:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local uuid_var_name
2024-05-08T01:15:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:15:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local error_code
2024-05-08T01:15:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local peer_uuid
2024-05-08T01:15:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:15:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:15:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2
2024-05-08T01:15:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ peer_uuid=8fda577f-a20d-47c9-89a8-9e056ea57c49
2024-05-08T01:15:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=0
2024-05-08T01:15:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:15:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -eq 17 ']'
2024-05-08T01:15:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 0 -eq 0
2024-05-08T01:15:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:15:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:15:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ peer_add cluster1 mirror_parent cluster2
2024-05-08T01:15:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:15:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:15:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local client_cluster=cluster2
2024-05-08T01:15:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local remote_cluster=cluster2
2024-05-08T01:15:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local uuid_var_name
2024-05-08T01:15:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:15:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local error_code
2024-05-08T01:15:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local peer_uuid
2024-05-08T01:15:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:15:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:15:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2
2024-05-08T01:15:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ peer_uuid=d67a7ec3-c140-49f2-84db-ae12a16e75f9
2024-05-08T01:15:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=0
2024-05-08T01:15:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:15:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -eq 17 ']'
2024-05-08T01:15:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 0 -eq 0
2024-05-08T01:15:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:15:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:15:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ setup_pools cluster2 cluster1
2024-05-08T01:15:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:15:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local remote_cluster=cluster1
2024-05-08T01:15:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mon_map_file
2024-05-08T01:15:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mon_addr
2024-05-08T01:15:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local admin_key_file
2024-05-08T01:15:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local uuid
2024-05-08T01:15:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:15:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64
2024-05-08T01:15:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:pool 'mirror' already exists
2024-05-08T01:15:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:15:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64
2024-05-08T01:15:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:pool 'mirror_parent' already exists
2024-05-08T01:15:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:15:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 pool init mirror
2024-05-08T01:15:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:15:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 pool init mirror_parent
2024-05-08T01:15:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:15:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:15:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror pool
2024-05-08T01:15:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image
2024-05-08T01:15:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1
2024-05-08T01:15:49.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2
2024-05-08T01:15:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 pool
2024-05-08T01:15:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image
2024-05-08T01:15:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -z ']'
2024-05-08T01:15:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -z ']'
2024-05-08T01:15:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ peer_add cluster2 mirror cluster1
2024-05-08T01:15:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:15:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local client_cluster=cluster1
2024-05-08T01:15:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local remote_cluster=cluster1
2024-05-08T01:15:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local uuid_var_name
2024-05-08T01:15:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:15:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local error_code
2024-05-08T01:15:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local peer_uuid
2024-05-08T01:15:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:15:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:15:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1
2024-05-08T01:15:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ peer_uuid=8feb6fcc-11e0-4bb3-a726-e9c13294c777
2024-05-08T01:15:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=0
2024-05-08T01:15:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:15:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -eq 17 ']'
2024-05-08T01:15:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 0 -eq 0
2024-05-08T01:15:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ peer_add cluster2 mirror_parent cluster1
2024-05-08T01:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local client_cluster=cluster1
2024-05-08T01:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local remote_cluster=cluster1
2024-05-08T01:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local uuid_var_name
2024-05-08T01:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local error_code
2024-05-08T01:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local peer_uuid
2024-05-08T01:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:15:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:15:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1
2024-05-08T01:15:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ peer_uuid=cd45e267-4b36-4e8c-beb7-48d2f1898243
2024-05-08T01:15:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=0
2024-05-08T01:15:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:15:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -eq 17 ']'
2024-05-08T01:15:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 0 -eq 0
2024-05-08T01:15:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:15:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:15:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:15:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: add image and test replay'
2024-05-08T01:15:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:15:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:15:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:15:49 TEST: add image and test replay
2024-05-08T01:15:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:15:49 TEST: add image and test replay
2024-05-08T01:15:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirrors cluster1
2024-05-08T01:15:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:15:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:15:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:15:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster1:0
2024-05-08T01:15:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:15:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:15:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:15:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:15:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:15:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:15:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:15:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:15:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:15:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:15:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:15:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:15:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:15:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:15:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:15:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:15:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster1:1
2024-05-08T01:15:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:15:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:15:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:15:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:1
2024-05-08T01:15:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:15:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:15:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:15:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:15:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster1:1 ']'
2024-05-08T01:15:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:15:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:15:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:15:49.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:15:49.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:15:49.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:15:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:15:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster1:2
2024-05-08T01:15:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:15:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:15:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:15:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:2
2024-05-08T01:15:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:15:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:15:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:15:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:15:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster1:2 ']'
2024-05-08T01:15:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:15:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:15:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:15:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:15:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:15:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:15:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:15:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster1:3
2024-05-08T01:15:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:15:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:15:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:15:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:3
2024-05-08T01:15:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:15:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:15:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:15:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:15:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster1:3 ']'
2024-05-08T01:15:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:15:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:15:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:15:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:15:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:15:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:15:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image=test
2024-05-08T01:15:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image_and_enable_mirror cluster2 mirror test journal
2024-05-08T01:15:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:15:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:15:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n journal ']'
2024-05-08T01:15:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image cluster2 mirror test
2024-05-08T01:15:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:15:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=128
2024-05-08T01:15:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:15:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test
2024-05-08T01:15:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ is_pool_mirror_mode_image mirror
2024-05-08T01:15:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' pool = image ']'
2024-05-08T01:15:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ case "${pool}" in
2024-05-08T01:15:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:15:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_image_meta cluster2 mirror test key1 value1
2024-05-08T01:15:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:15:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local key=key1
2024-05-08T01:15:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=value1
2024-05-08T01:15:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 image-meta set mirror/test key1 value1
2024-05-08T01:15:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_image_meta cluster2 mirror test key2 value2
2024-05-08T01:15:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:15:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local key=key2
2024-05-08T01:15:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=value2
2024-05-08T01:15:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 image-meta set mirror/test key2 value2
2024-05-08T01:15:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:15:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:15:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:15:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:15:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:15:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:15:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:15:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:15:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:15:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:15:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:15:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:15:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:15:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:15:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:15:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:15:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:15:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:15:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:15:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:15:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:15:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:15:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:15:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:15:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:15:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:15:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:15:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:15:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:15:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:15:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:15:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:15:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:15:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:15:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:15:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:15:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:15:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:15:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:15:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:15:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:15:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:15:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:15:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:15:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:15:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:15:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:15:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:15:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:15:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:15:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:15:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:15:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:15:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:15:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:15:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:15:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:15:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:15:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:15:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:15:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:15:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:15:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:15:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:15:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test
2024-05-08T01:15:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:15:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:15:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:15:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:15:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:15:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:15:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:15:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:15:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:15:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:15:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:15:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:15:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:15:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:15:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:15:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:15:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:15:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:15:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:15:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:15:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test
2024-05-08T01:15:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:15:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:15:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:15:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:15:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test
2024-05-08T01:15:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:15:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:15:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:15:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:15:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:15:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:15:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:15:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:15:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:15:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:15:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:15:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:15:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:15:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:15:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:15:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:15:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:15:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:15:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:15:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:15:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:15:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:15:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:15:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:15:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:15:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:15:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:15:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:15:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:15:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:15:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:15:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:15:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:15:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test
2024-05-08T01:15:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:15:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:15:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test
2024-05-08T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:15:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:15:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:15:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:15:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:15:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:15:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:15:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:15:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:15:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:15:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:15:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:15:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:15:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:15:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:15:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:15:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:15:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:15:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:15:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:15:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:15:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:15:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:15:51.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:15:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:15:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:15:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test
2024-05-08T01:15:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:15:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:15:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:15:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:15:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 1
2024-05-08T01:15:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:15:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:15:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:15:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:15:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:15:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:15:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:15:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:15:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:15:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:15:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:15:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:15:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:15:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:15:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:15:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:15:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:15:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:15:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:15:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:15:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:15:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:15:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:15:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:15:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:15:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:15:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:15:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:15:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:15:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:15:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:15:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:15:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:15:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:15:52.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:15:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:15:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:15:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:15:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:15:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:15:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:15:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:15:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test
2024-05-08T01:15:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:15:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:15:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:15:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:15:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:15:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:15:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:15:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:15:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:15:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:15:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:15:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:15:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:15:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:15:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:15:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:15:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:15:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:15:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:15:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:15:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:15:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:15:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:15:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:15:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:15:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:15:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:15:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:15:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:15:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:15:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:15:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:15:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:15:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test
2024-05-08T01:15:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:15:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:15:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:15:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:15:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test
2024-05-08T01:15:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:15:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:15:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:15:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:15:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:15:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:15:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:15:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:15:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:15:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:15:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:15:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:15:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:15:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:15:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:15:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:15:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:15:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:15:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:15:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:15:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test
2024-05-08T01:15:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:15:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:15:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:15:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:15:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test
2024-05-08T01:15:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:15:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:15:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:15:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:15:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:15:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:15:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:15:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:15:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:15:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:15:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:15:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:15:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:15:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:15:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:15:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:15:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:15:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:15:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:15:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:15:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:15:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:15:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:15:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:15:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:15:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:15:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:15:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:15:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:15:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:15:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:15:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:15:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:15:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test
2024-05-08T01:15:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:15:52.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:15:52.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:15:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:15:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 2
2024-05-08T01:15:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:15:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:15:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:15:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:15:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:15:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:15:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:15:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:15:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:15:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:15:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:15:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:15:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:15:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:15:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:15:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:15:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:15:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:15:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:15:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:15:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:15:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:15:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:15:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:15:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:15:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:15:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:15:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:15:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:15:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:15:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:15:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:15:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:15:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:15:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:15:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:15:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:15:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:15:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:15:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:15:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:15:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:15:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:15:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:15:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:15:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:15:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test 100
2024-05-08T01:15:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:15:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:15:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:15:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:15:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:15:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:15:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:15:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:15:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2083.38   bytes/sec: 8.1 MiB/s
2024-05-08T01:15:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:15:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:15:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:15:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:15:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:15:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:15:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:15:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:15:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:15:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:15:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:15:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:15:55.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:15:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:15:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:15:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:15:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:15:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:15:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:15:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:15:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:15:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:15:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:15:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:15:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:15:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:15:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:15:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:15:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:15:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:15:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:15:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:15:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:15:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:15:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:15:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:15:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:15:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:15:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:15:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:15:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:15:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:15:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:15:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:15:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:15:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:15:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:15:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:15:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:15:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:15:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:15:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:15:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:15:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:15:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:15:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:15:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:15:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:15:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:15:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:15:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:15:55.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:15:55.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:15:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:15:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:15:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:15:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:15:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:15:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:15:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:15:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:15:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:15:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:15:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:15:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:15:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:15:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:15:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:15:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:15:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:15:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected]
2024-05-08T01:15:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=4, entry_tid=92], [object_number=3, tag_tid=4, entry_tid=91], [object_number=2, tag_tid=4, entry_tid=90], [object_number=1, tag_tid=4, entry_tid=89]]], state=connected]
2024-05-08T01:15:55.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:15:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=4, entry_tid=101'
2024-05-08T01:15:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:15:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:15:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:15:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:15:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:15:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:15:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:15:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:15:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:15:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:15:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:15:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:15:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:15:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:15:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:15:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:15:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:15:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:15:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected]
2024-05-08T01:15:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=4, entry_tid=92], [object_number=3, tag_tid=4, entry_tid=91], [object_number=2, tag_tid=4, entry_tid=90], [object_number=1, tag_tid=4, entry_tid=89]]], state=connected]
2024-05-08T01:15:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:15:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=4, entry_tid=92'
2024-05-08T01:15:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=4, entry_tid=101' -a 'object_number=1, tag_tid=4, entry_tid=101' = 'object_number=0, tag_tid=4, entry_tid=92'
2024-05-08T01:15:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=4, entry_tid=92' '!=' ''
2024-05-08T01:15:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:15:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=4, entry_tid=92' = ''
2024-05-08T01:15:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=0, tag_tid=4, entry_tid=92'
2024-05-08T01:15:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=4, entry_tid=101'
2024-05-08T01:15:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:15:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:15:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=4
2024-05-08T01:15:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=4, entry_tid=92'
2024-05-08T01:15:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:15:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:15:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=4
2024-05-08T01:15:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=4, entry_tid=101'
2024-05-08T01:15:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:15:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:15:55.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=101
2024-05-08T01:15:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=4, entry_tid=92'
2024-05-08T01:15:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:15:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:15:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=92
2024-05-08T01:15:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 4 = 4 -a 101 -le 92
2024-05-08T01:15:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:15:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:15:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:15:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:15:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:15:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:15:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:15:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:15:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:15:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:15:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:15:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:15:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:15:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:15:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:15:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:15:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:15:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:15:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:15:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:15:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:15:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:15:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:15:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:15:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:15:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:15:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:15:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:15:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:15:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:15:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:15:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:15:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:15:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:15:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:15:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:15:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:15:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:15:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:15:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:15:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:15:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:15:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:15:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:15:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:15:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:15:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:15:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:15:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:15:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:15:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:15:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:15:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:15:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:15:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:15:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:15:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:15:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:15:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:15:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:15:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:15:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:15:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:15:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:15:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:15:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:15:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:15:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:15:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:15:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:15:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:15:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:15:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected]
2024-05-08T01:15:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected]
2024-05-08T01:15:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:15:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=4, entry_tid=101'
2024-05-08T01:15:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:15:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:15:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:15:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:15:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:15:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:15:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:15:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:15:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:15:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:15:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:15:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:15:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:15:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:15:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:15:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:15:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:15:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:15:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected]
2024-05-08T01:15:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected]
2024-05-08T01:15:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:15:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=1, tag_tid=4, entry_tid=101'
2024-05-08T01:15:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=4, entry_tid=101' -a 'object_number=1, tag_tid=4, entry_tid=101' = 'object_number=1, tag_tid=4, entry_tid=101'
2024-05-08T01:15:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:15:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test
2024-05-08T01:15:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:15:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:15:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position
2024-05-08T01:15:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:15:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:15:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:15:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:15:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:15:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:15:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position ''
2024-05-08T01:15:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:15:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:15:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:15:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:15:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:15:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:15:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:15:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:15:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   757bbca6-4715-40fb-ae9f-f4772f7276b1
2024-05-08T01:15:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:15:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}}
2024-05-08T01:15:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:15:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:15:54'
2024-05-08T01:15:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:15:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:15:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:15:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}}
2024-05-08T01:15:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:15:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:15:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:15:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:15:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:15:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:15:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:15:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:15:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:15:54'
2024-05-08T01:15:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:15:54'
2024-05-08T01:15:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:15:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:15:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:15:54'
2024-05-08T01:15:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:15:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:15:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:15:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4254</service_id><instance_id>4265</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4258</service_id><instance_id>4270</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4264</service_id><instance_id>4276</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4272</service_id><instance_id>4277</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:1,&quot;object_number&quot;:1,&quot;tag_tid&quot;:2},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:3}}</description><daemon_service><service_id>4254</service_id><instance_id>4265</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:15:54</last_update></image></images></status>'
2024-05-08T01:15:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:15:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:15:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:15:54'
2024-05-08T01:15:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:15:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:15:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:15:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:15:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:15:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:15:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:15:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -z '' ']'
2024-05-08T01:15:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test down+unknown
2024-05-08T01:15:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:15:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=down+unknown
2024-05-08T01:15:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:15:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:15:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:15:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:15:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test down+unknown '' ''
2024-05-08T01:15:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:15:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=down+unknown
2024-05-08T01:15:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:15:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:15:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:15:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:15:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:15:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:15:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   757bbca6-4715-40fb-ae9f-f4772f7276b1
2024-05-08T01:15:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       down+unknown
2024-05-08T01:15:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: status not found
2024-05-08T01:15:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update:
2024-05-08T01:15:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:15:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:15:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:15:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}}
2024-05-08T01:15:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:15:54'
2024-05-08T01:15:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*down+unknown'
2024-05-08T01:15:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       down+unknown
2024-05-08T01:15:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:15:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: status not found
2024-05-08T01:15:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:15:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo down+unknown
2024-05-08T01:15:58.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:15:58.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: '
2024-05-08T01:15:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:15:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:15:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update=
2024-05-08T01:15:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test down+unknown ''
2024-05-08T01:15:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:15:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=down+unknown
2024-05-08T01:15:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local prev_last_update=
2024-05-08T01:15:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:15:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:15:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:15:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>UNKNOWN</health><daemon_health>UNKNOWN</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>down+unknown</state><description>status not found</description><last_update></last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:1,&quot;object_number&quot;:1,&quot;tag_tid&quot;:2},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:3}}</description><last_update>2024-05-08 01:15:54</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:15:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:15:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:15:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update=
2024-05-08T01:15:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:15:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=down+unknown
2024-05-08T01:15:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo down+unknown
2024-05-08T01:15:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep down+unknown
2024-05-08T01:15:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:down+unknown
2024-05-08T01:15:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:15:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:15:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror test
2024-05-08T01:15:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:15:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror-test.export
2024-05-08T01:15:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror-test.export
2024-05-08T01:15:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:15:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.export
2024-05-08T01:15:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror-test.export
2024-05-08T01:15:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror-test.export
2024-05-08T01:15:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:15:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:15:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:15:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export
2024-05-08T01:15:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:15:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:15:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:15:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:15:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:15:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:15:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_image_meta cluster1 mirror test key1 value1
2024-05-08T01:15:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:15:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local key=key1
2024-05-08T01:15:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local value=value1
2024-05-08T01:15:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 image-meta get mirror/test key1
2024-05-08T01:15:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test value1 = value1
2024-05-08T01:15:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_image_meta cluster1 mirror test key2 value2
2024-05-08T01:15:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:15:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:15:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:15:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local key=key2
2024-05-08T01:15:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local value=value2
2024-05-08T01:15:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 image-meta get mirror/test key2
2024-05-08T01:15:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test value2 = value2
2024-05-08T01:15:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: stop mirror, add image, start mirror and test replay'
2024-05-08T01:15:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:15:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:15:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:15:59 TEST: stop mirror, add image, start mirror and test replay
2024-05-08T01:15:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:15:59 TEST: stop mirror, add image, start mirror and test replay
2024-05-08T01:15:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirrors cluster1
2024-05-08T01:15:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:15:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:15:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:15:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:15:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster1:0 ''
2024-05-08T01:15:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:15:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:15:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:15:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:15:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster1:0
2024-05-08T01:15:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1:0
2024-05-08T01:15:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:15:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:15:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:15:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:15:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:15:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:15:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:15:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:15:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster1
2024-05-08T01:15:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster1
2024-05-08T01:15:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:15:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:15:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file'
2024-05-08T01:15:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid
2024-05-08T01:15:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid
2024-05-08T01:15:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=30275
2024-05-08T01:15:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 30275 ']'
2024-05-08T01:15:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 30275
2024-05-08T01:15:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:15:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:16:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=30275 '$2 == pid {print; exit 1}'
2024-05-08T01:16:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:16:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:16:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=30275 '$2 == pid {print; exit 1}'
2024-05-08T01:16:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:16:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster2
2024-05-08T01:16:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:16:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster1:0
2024-05-08T01:16:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file'
2024-05-08T01:16:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid
2024-05-08T01:16:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid
2024-05-08T01:16:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster1:1 ''
2024-05-08T01:16:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:16:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:16:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:16:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:16:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster1:1
2024-05-08T01:16:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1:1
2024-05-08T01:16:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:16:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:16:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:16:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:16:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster1
2024-05-08T01:16:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster1
2024-05-08T01:16:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:16:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:16:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file'
2024-05-08T01:16:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid
2024-05-08T01:16:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid
2024-05-08T01:16:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=30303
2024-05-08T01:16:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 30303 ']'
2024-05-08T01:16:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 30303
2024-05-08T01:16:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:16:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:16:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=30303 '$2 == pid {print; exit 1}'
2024-05-08T01:16:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:16:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:16:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=30303 '$2 == pid {print; exit 1}'
2024-05-08T01:16:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:16:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:16:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:16:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:16:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:16:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:16:01.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:16:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster2
2024-05-08T01:16:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:16:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:16:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:16:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:16:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:16:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:16:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster1:1
2024-05-08T01:16:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:16:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:16:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:16:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:16:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:16:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file'
2024-05-08T01:16:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid
2024-05-08T01:16:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid
2024-05-08T01:16:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster1:2 ''
2024-05-08T01:16:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:16:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:16:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:16:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:16:01.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster1:2
2024-05-08T01:16:01.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1:2
2024-05-08T01:16:01.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:01.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:16:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:16:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:16:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:16:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster1
2024-05-08T01:16:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster1
2024-05-08T01:16:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:16:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:16:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file'
2024-05-08T01:16:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid
2024-05-08T01:16:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid
2024-05-08T01:16:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=30331
2024-05-08T01:16:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 30331 ']'
2024-05-08T01:16:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 30331
2024-05-08T01:16:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:16:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:16:02.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=30331 '$2 == pid {print; exit 1}'
2024-05-08T01:16:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:16:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:16:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=30331 '$2 == pid {print; exit 1}'
2024-05-08T01:16:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:16:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:16:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:16:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:16:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:16:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:16:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:16:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster2
2024-05-08T01:16:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:16:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:16:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:16:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:16:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:16:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:16:02.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:02.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster1:2
2024-05-08T01:16:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:16:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:16:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:16:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:16:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:16:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file'
2024-05-08T01:16:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid
2024-05-08T01:16:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid
2024-05-08T01:16:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster1:3 ''
2024-05-08T01:16:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:16:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:16:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:16:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:16:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster1:3
2024-05-08T01:16:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1:3
2024-05-08T01:16:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:16:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:16:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:16:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:16:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster1
2024-05-08T01:16:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster1
2024-05-08T01:16:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:16:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:16:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file'
2024-05-08T01:16:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid
2024-05-08T01:16:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid
2024-05-08T01:16:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=30406
2024-05-08T01:16:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 30406 ']'
2024-05-08T01:16:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 30406
2024-05-08T01:16:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:16:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:03.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:16:03.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=30406 '$2 == pid {print; exit 1}'
2024-05-08T01:16:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:16:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:16:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=30406 '$2 == pid {print; exit 1}'
2024-05-08T01:16:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:16:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:16:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:16:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:16:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:16:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:16:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:16:03.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster2
2024-05-08T01:16:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:16:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:16:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:16:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:16:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:16:03.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:16:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster1:3
2024-05-08T01:16:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:16:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:16:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:16:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:16:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:16:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file'
2024-05-08T01:16:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid
2024-05-08T01:16:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid
2024-05-08T01:16:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image1=test1
2024-05-08T01:16:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image_and_enable_mirror cluster2 mirror test1 journal
2024-05-08T01:16:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:16:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:16:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n journal ']'
2024-05-08T01:16:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image cluster2 mirror test1
2024-05-08T01:16:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:16:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=128
2024-05-08T01:16:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:16:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test1
2024-05-08T01:16:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ is_pool_mirror_mode_image mirror
2024-05-08T01:16:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' pool = image ']'
2024-05-08T01:16:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ case "${pool}" in
2024-05-08T01:16:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:16:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test1 100
2024-05-08T01:16:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:16:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:16:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:16:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:16:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:16:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test1 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:16:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:16:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:16:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2500.04   bytes/sec: 9.8 MiB/s
2024-05-08T01:16:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirrors cluster1
2024-05-08T01:16:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:16:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster1:0
2024-05-08T01:16:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:16:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:16:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:16:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:16:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:16:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:16:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster1:1
2024-05-08T01:16:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:16:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:16:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:1
2024-05-08T01:16:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:16:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:16:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:16:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:16:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster1:2
2024-05-08T01:16:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:16:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:16:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:2
2024-05-08T01:16:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:16:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:16:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:16:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:16:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster1:3
2024-05-08T01:16:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:16:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:16:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:3
2024-05-08T01:16:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:16:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:16:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:16:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:16:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test1
2024-05-08T01:16:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started
2024-05-08T01:16:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:16:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test1 started
2024-05-08T01:16:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:16:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:16:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:16:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1
2024-05-08T01:16:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:16:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:16:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:16:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:16:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:16:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1
2024-05-08T01:16:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1
2024-05-08T01:16:05.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test1']
2024-05-08T01:16:05.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:16:05.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:16:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test1
2024-05-08T01:16:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:16:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:16:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:16:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:16:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:16:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:16:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:16:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:16:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:16:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:16:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:16:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:16:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:16:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test1
2024-05-08T01:16:05.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test1']
2024-05-08T01:16:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:16:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:16:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test1
2024-05-08T01:16:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:16:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:16:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:16:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:16:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:16:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:16:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:16:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:16:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:16:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:16:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:16:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:16:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:16:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test1
2024-05-08T01:16:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test1']
2024-05-08T01:16:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:16:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:16:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test1
2024-05-08T01:16:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:16:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:16:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:16:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:16:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:16:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:16:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:16:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:16:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:16:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:16:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:16:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:16:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:16:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test1
2024-05-08T01:16:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test1']
2024-05-08T01:16:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:16:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:16:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 1
2024-05-08T01:16:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1
2024-05-08T01:16:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1
2024-05-08T01:16:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test1']
2024-05-08T01:16:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:16:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:16:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test1
2024-05-08T01:16:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:16:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:16:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:16:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:16:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:16:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:16:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:16:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:16:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:16:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:16:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:16:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:16:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:16:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test1
2024-05-08T01:16:07.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test1']
2024-05-08T01:16:07.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:16:07.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:16:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test1
2024-05-08T01:16:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:16:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:16:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:16:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:16:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:16:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:16:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:16:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:16:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:16:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:16:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:16:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:16:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:16:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test1
2024-05-08T01:16:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test1']
2024-05-08T01:16:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:16:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:16:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test1
2024-05-08T01:16:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:16:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:16:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:16:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:16:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:16:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:16:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:16:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:16:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:16:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:16:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:16:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:16:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:16:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test1
2024-05-08T01:16:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test1']
2024-05-08T01:16:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:16:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:16:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 2
2024-05-08T01:16:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1
2024-05-08T01:16:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1
2024-05-08T01:16:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:16:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:16:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:16:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:16:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:16:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:16:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:16:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test1
2024-05-08T01:16:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:16:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:16:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:16:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test1
2024-05-08T01:16:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:16:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:16:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:16:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:16:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:16:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:16:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:16:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test1
2024-05-08T01:16:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:16:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test1 ']'
2024-05-08T01:16:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test1'
2024-05-08T01:16:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test1
2024-05-08T01:16:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:16:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:16:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:16:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:16:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:16:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test1
2024-05-08T01:16:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:16:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:16:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:16:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:16:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:16:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test1
2024-05-08T01:16:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test1
2024-05-08T01:16:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:16:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test1
2024-05-08T01:16:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test1 ''
2024-05-08T01:16:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:16:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test1
2024-05-08T01:16:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:16:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test1.status
2024-05-08T01:16:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test1.status
2024-05-08T01:16:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:16:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test1.status
2024-05-08T01:16:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test1
2024-05-08T01:16:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test1.status
2024-05-08T01:16:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:16:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:16:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:16:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:16:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:16:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test1.status
2024-05-08T01:16:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:16:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test1
2024-05-08T01:16:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:16:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test1
2024-05-08T01:16:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test1 '..*'
2024-05-08T01:16:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:16:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test1
2024-05-08T01:16:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:16:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test1.status
2024-05-08T01:16:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test1.status
2024-05-08T01:16:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:16:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test1.status
2024-05-08T01:16:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test1
2024-05-08T01:16:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test1.status
2024-05-08T01:16:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:16:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:16:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:16:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:16:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:16:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test1.status
2024-05-08T01:16:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:16:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:16:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test1
2024-05-08T01:16:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:16:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying primary_position
2024-05-08T01:16:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:16:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying primary_position ''
2024-05-08T01:16:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:16:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1
2024-05-08T01:16:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test1:
2024-05-08T01:16:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   4d17c619-1bc7-4c02-b113-0e958e942b07
2024-05-08T01:16:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:16:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:16:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:16:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:16:07'
2024-05-08T01:16:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:16:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:16:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:16:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:16:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:16:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:16:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:16:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:16:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:16:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:16:11.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:07'
2024-05-08T01:16:11.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2024-05-08 01:16:07'
2024-05-08T01:16:11.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:11.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:11.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:11.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:11.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:16:07'
2024-05-08T01:16:11.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:16:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:4},&quot;primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:4}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:08</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:1},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:07</last_update></image></images></status>'
2024-05-08T01:16:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:16:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update'
2024-05-08T01:16:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:07'
2024-05-08T01:16:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state'
2024-05-08T01:16:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:16:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:16:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -z '' ']'
2024-05-08T01:16:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test1 down+unknown
2024-05-08T01:16:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:16:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=down+unknown
2024-05-08T01:16:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test1 down+unknown '' ''
2024-05-08T01:16:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:16:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=down+unknown
2024-05-08T01:16:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test1
2024-05-08T01:16:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test1:
2024-05-08T01:16:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   4d17c619-1bc7-4c02-b113-0e958e942b07
2024-05-08T01:16:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       down+unknown
2024-05-08T01:16:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: status not found
2024-05-08T01:16:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update:
2024-05-08T01:16:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:16:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:16:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:16:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:16:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:16:07'
2024-05-08T01:16:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*down+unknown'
2024-05-08T01:16:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       down+unknown
2024-05-08T01:16:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:16:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: status not found
2024-05-08T01:16:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:16:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo down+unknown
2024-05-08T01:16:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:16:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: '
2024-05-08T01:16:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:16:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:16:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update=
2024-05-08T01:16:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test1 down+unknown ''
2024-05-08T01:16:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:16:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=down+unknown
2024-05-08T01:16:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local prev_last_update=
2024-05-08T01:16:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:16:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>UNKNOWN</health><daemon_health>UNKNOWN</daemon_health><image_health>OK</image_health><states><replaying>2</replaying></states></summary><daemons></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>down+unknown</state><description>status not found</description><last_update></last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:4},&quot;primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:4}}</description><last_update>2024-05-08 01:16:08</last_update></peer_site></peer_sites></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>down+unknown</state><description>status not found</description><last_update></last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:1},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><last_update>2024-05-08 01:16:07</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:16:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:16:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update'
2024-05-08T01:16:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update=
2024-05-08T01:16:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state'
2024-05-08T01:16:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=down+unknown
2024-05-08T01:16:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo down+unknown
2024-05-08T01:16:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep down+unknown
2024-05-08T01:16:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:down+unknown
2024-05-08T01:16:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror test1
2024-05-08T01:16:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:16:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror-test1.export
2024-05-08T01:16:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror-test1.export
2024-05-08T01:16:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:16:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror-test1.export
2024-05-08T01:16:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror-test1.export
2024-05-08T01:16:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror-test1.export
2024-05-08T01:16:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:16:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror-test1.export
2024-05-08T01:16:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test1.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test1.export
2024-05-08T01:16:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/test1 /tmp/tmp.3xddryFtVF/cluster2-mirror-test1.export
2024-05-08T01:16:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:16:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/test1 /tmp/tmp.3xddryFtVF/cluster1-mirror-test1.export
2024-05-08T01:16:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:16:12.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror-test1.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test1.export
2024-05-08T01:16:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test1.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test1.export
2024-05-08T01:16:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: test the first image is replaying after restart'
2024-05-08T01:16:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:16:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:16:12.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:16:12 TEST: test the first image is replaying after restart
2024-05-08T01:16:12.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:16:12 TEST: test the first image is replaying after restart
2024-05-08T01:16:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test 100
2024-05-08T01:16:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:16:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:16:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:16:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:16:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:16:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:16:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:16:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:16:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2500.04   bytes/sec: 9.8 MiB/s
2024-05-08T01:16:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:16:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:16:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:16:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:16:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:16:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:16:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:16:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:16:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:16:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:16:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:16:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:16:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:16:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:16:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:16:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:16:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:16:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:16:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:16:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:16:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:16:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:16:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:16:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:16:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:16:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:16:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:16:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:16:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:16:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:16:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:16:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:16:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:16:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:16:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:16:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:16:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:16:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:16:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:16:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:16:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:16:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:16:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:16:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:16:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:16:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:16:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:16:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:16:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:16:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:16:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:16:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:16:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:16:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:16:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:16:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:16:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:16:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:16:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:16:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:16:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:16:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:16:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:16:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:16:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:16:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:16:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:16:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:16:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected]
2024-05-08T01:16:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:16:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:16:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:16:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:16:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:16:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:16:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:16:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:16:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:16:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:16:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:16:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:16:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:16:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:16:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:16:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:16:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:16:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:16:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:16:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected]
2024-05-08T01:16:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:16:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=1, tag_tid=4, entry_tid=101'
2024-05-08T01:16:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=4, entry_tid=101'
2024-05-08T01:16:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=1, tag_tid=4, entry_tid=101' '!=' ''
2024-05-08T01:16:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:16:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=1, tag_tid=4, entry_tid=101' = ''
2024-05-08T01:16:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=1, tag_tid=4, entry_tid=101'
2024-05-08T01:16:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:16:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:16:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:16:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=5
2024-05-08T01:16:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=4, entry_tid=101'
2024-05-08T01:16:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:16:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:16:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=4
2024-05-08T01:16:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:16:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:16:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:16:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=101
2024-05-08T01:16:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=4, entry_tid=101'
2024-05-08T01:16:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:16:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:16:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=101
2024-05-08T01:16:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 5 = 4 -a 101 -le 101
2024-05-08T01:16:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:16:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:16:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:16:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:16:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:16:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:16:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:16:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:16:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:16:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:16:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:16:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:16:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:16:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:16:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:16:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:16:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:16:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:16:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:16:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:16:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:16:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:16:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:16:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:16:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:16:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:16:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:16:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:16:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:16:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:16:14.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:16:14.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:16:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:16:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:16:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:16:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:16:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:16:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=2, tag_tid=5, entry_tid=90], [object_number=1, tag_tid=5, entry_tid=89], [object_number=0, tag_tid=5, entry_tid=88], [object_number=3, tag_tid=5, entry_tid=87]]], state=connected]
2024-05-08T01:16:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:16:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:16:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:16:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:16:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:16:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:16:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:16:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:16:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:16:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:16:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:16:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:16:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:16:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:16:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:16:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:16:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:16:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:16:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:16:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=2, tag_tid=5, entry_tid=90], [object_number=1, tag_tid=5, entry_tid=89], [object_number=0, tag_tid=5, entry_tid=88], [object_number=3, tag_tid=5, entry_tid=87]]], state=connected]
2024-05-08T01:16:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:16:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=2, tag_tid=5, entry_tid=90'
2024-05-08T01:16:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=2, tag_tid=5, entry_tid=90'
2024-05-08T01:16:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=2, tag_tid=5, entry_tid=90' '!=' 'object_number=1, tag_tid=4, entry_tid=101'
2024-05-08T01:16:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:16:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=2, tag_tid=5, entry_tid=90' = 'object_number=1, tag_tid=4, entry_tid=101'
2024-05-08T01:16:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=2, tag_tid=5, entry_tid=90'
2024-05-08T01:16:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:16:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:16:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:16:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=5
2024-05-08T01:16:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=2, tag_tid=5, entry_tid=90'
2024-05-08T01:16:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:16:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:16:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=5
2024-05-08T01:16:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:16:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:16:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:16:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=101
2024-05-08T01:16:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=2, tag_tid=5, entry_tid=90'
2024-05-08T01:16:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:16:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:16:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=90
2024-05-08T01:16:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 5 = 5 -a 101 -le 90
2024-05-08T01:16:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:16:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:16:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:16:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:16:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:16:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:16:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:16:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:16:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:16:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:16:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:16:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:16:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:16:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:16:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:16:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:16:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:16:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:16:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:16:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:16:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:16:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:16:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:16:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:16:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:16:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:16:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:16:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:16:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:16:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:16:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:16:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:16:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:16:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:16:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:16:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:16:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:16:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:16:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:16:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:16:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:16:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:16:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:16:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:16:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:16:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:16:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:16:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:16:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:16:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:16:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:16:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:16:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test
2024-05-08T01:16:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:16:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position
2024-05-08T01:16:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:16:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position ''
2024-05-08T01:16:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:16:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:16:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:16:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   757bbca6-4715-40fb-ae9f-f4772f7276b1
2024-05-08T01:16:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:16:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4}}
2024-05-08T01:16:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:16:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:16:08'
2024-05-08T01:16:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:16:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:16:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:16:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4}}
2024-05-08T01:16:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:16:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:16:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:16:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:16:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:16:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:16:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:08'
2024-05-08T01:16:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:16:08'
2024-05-08T01:16:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:16:08'
2024-05-08T01:16:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:16:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:4},&quot;primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:4}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:08</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:1},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:07</last_update></image></images></status>'
2024-05-08T01:16:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:16:16.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:16:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:08'
2024-05-08T01:16:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:16:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:16:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:16:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror test
2024-05-08T01:16:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:16:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror-test.export
2024-05-08T01:16:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror-test.export
2024-05-08T01:16:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:16:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.export
2024-05-08T01:16:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror-test.export
2024-05-08T01:16:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror-test.export
2024-05-08T01:16:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:16:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:16:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:16:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export
2024-05-08T01:16:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:16:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:16:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:16:16.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:16:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:16:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -z '' ']'
2024-05-08T01:16:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: stop/start/restart mirror via admin socket'
2024-05-08T01:16:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:16:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:16:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:16:16 TEST: stop/start/restart mirror via admin socket
2024-05-08T01:16:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:16:16 TEST: stop/start/restart mirror via admin socket
2024-05-08T01:16:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ all_admin_daemons cluster1 rbd mirror stop
2024-05-08T01:16:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:16:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror stop
2024-05-08T01:16:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:16:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:16:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:16:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:16:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:16:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop
2024-05-08T01:16:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:1 rbd mirror stop
2024-05-08T01:16:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:16:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:16:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:1
2024-05-08T01:16:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:16:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:16:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:16:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:16:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:16:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:16:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:16:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:16:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:16:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop
2024-05-08T01:16:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:2 rbd mirror stop
2024-05-08T01:16:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:16:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:16:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:2
2024-05-08T01:16:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:16:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:16:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:16:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:16:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:16:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:16:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:16:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:16:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:16:17.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:17.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:17.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:17.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror stop
2024-05-08T01:16:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:3 rbd mirror stop
2024-05-08T01:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:3
2024-05-08T01:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:16:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:16:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:16:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:16:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:16:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:16:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:16:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror stop
2024-05-08T01:16:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:16:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:16:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:16:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:16:18.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:18.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:18.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:18.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:16:18.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:16:18.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:16:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:16:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:16:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:16:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:16:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:16:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:16:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:16:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:16:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:16:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:16:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:16:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:16:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:16:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1
2024-05-08T01:16:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped
2024-05-08T01:16:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:16:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test1 stopped
2024-05-08T01:16:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:16:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:16:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:16:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1
2024-05-08T01:16:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:16:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:16:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:16:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:16:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:16:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1
2024-05-08T01:16:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1
2024-05-08T01:16:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:16:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:16:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:16:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:16:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:16:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:16:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:16:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:16:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:16:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:16:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:16:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   757bbca6-4715-40fb-ae9f-f4772f7276b1
2024-05-08T01:16:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:16:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:16:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:16:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:16:17'
2024-05-08T01:16:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:16:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:16:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:16:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: stopped
2024-05-08T01:16:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:16:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:16:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:16:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:16:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:16:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:16:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:16:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:16:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:17'
2024-05-08T01:16:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:16:17'
2024-05-08T01:16:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:16:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:16:17'
2024-05-08T01:16:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:16:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+stopped</state><description>stopped</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:17</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+stopped</state><description>stopped</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:17</last_update></image></images></status>'
2024-05-08T01:16:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:16:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:16:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:17'
2024-05-08T01:16:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:16:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:16:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:16:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:16:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:16:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped
2024-05-08T01:16:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:16:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' ''
2024-05-08T01:16:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:16:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1
2024-05-08T01:16:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test1:
2024-05-08T01:16:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   4d17c619-1bc7-4c02-b113-0e958e942b07
2024-05-08T01:16:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:16:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:16:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:16:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:16:17'
2024-05-08T01:16:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:16:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:16:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:16:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: stopped
2024-05-08T01:16:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:16:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:16:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:16:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:16:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:16:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:16:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:16:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:16:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:17'
2024-05-08T01:16:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2024-05-08 01:16:17'
2024-05-08T01:16:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:16:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:16:17'
2024-05-08T01:16:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:16:22.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+stopped</state><description>stopped</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:17</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+stopped</state><description>stopped</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:17</last_update></image></images></status>'
2024-05-08T01:16:22.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:16:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update'
2024-05-08T01:16:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:17'
2024-05-08T01:16:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state'
2024-05-08T01:16:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:16:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:16:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:16:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:16:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ all_admin_daemons cluster1 rbd mirror start
2024-05-08T01:16:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:16:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror start
2024-05-08T01:16:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:16:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:16:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:16:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:16:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:16:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start
2024-05-08T01:16:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:1 rbd mirror start
2024-05-08T01:16:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:16:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:16:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:1
2024-05-08T01:16:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:16:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:16:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:16:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:16:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:16:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:16:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:16:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror start
2024-05-08T01:16:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:2 rbd mirror start
2024-05-08T01:16:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:16:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:16:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:2
2024-05-08T01:16:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:16:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:16:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:16:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:16:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:16:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:16:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:16:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:16:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:16:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror start
2024-05-08T01:16:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:3 rbd mirror start
2024-05-08T01:16:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:16:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:16:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:3
2024-05-08T01:16:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:16:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:16:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:16:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:16:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:16:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:16:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:16:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:16:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:16:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror start
2024-05-08T01:16:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:16:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:16:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:16:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:16:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:16:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:16:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:16:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:16:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:16:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:16:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:16:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:16:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:16:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:16:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:23.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:23.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:16:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:16:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:16:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:16:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:16:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:16:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:16:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:16:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test1
2024-05-08T01:16:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started
2024-05-08T01:16:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:16:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test1 started
2024-05-08T01:16:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:16:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:16:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:16:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1
2024-05-08T01:16:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:16:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:16:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:16:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:16:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:16:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1
2024-05-08T01:16:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1
2024-05-08T01:16:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:16:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:16:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:16:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:16:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:16:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:16:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:16:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:16:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:16:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:16:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:16:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   757bbca6-4715-40fb-ae9f-f4772f7276b1
2024-05-08T01:16:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:16:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}
2024-05-08T01:16:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:16:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:16:22'
2024-05-08T01:16:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:16:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:16:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:16:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}
2024-05-08T01:16:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:16:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:16:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:16:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:16:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:16:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:16:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:22'
2024-05-08T01:16:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:16:22'
2024-05-08T01:16:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:16:22'
2024-05-08T01:16:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:16:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5},&quot;primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:22</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:22</last_update></image></images></status>'
2024-05-08T01:16:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:16:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:16:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:22'
2024-05-08T01:16:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:16:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:16:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:16:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying
2024-05-08T01:16:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' ''
2024-05-08T01:16:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1
2024-05-08T01:16:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test1:
2024-05-08T01:16:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   4d17c619-1bc7-4c02-b113-0e958e942b07
2024-05-08T01:16:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:16:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:16:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:16:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:16:22'
2024-05-08T01:16:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:16:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:16:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:16:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:16:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:16:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:16:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:16:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:16:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:16:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:16:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:22'
2024-05-08T01:16:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2024-05-08 01:16:22'
2024-05-08T01:16:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:16:22'
2024-05-08T01:16:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:16:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5},&quot;primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:22</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:22</last_update></image></images></status>'
2024-05-08T01:16:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:16:27.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update'
2024-05-08T01:16:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:22'
2024-05-08T01:16:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state'
2024-05-08T01:16:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:16:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:16:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ all_admin_daemons cluster1 rbd mirror restart
2024-05-08T01:16:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:16:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror restart
2024-05-08T01:16:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:16:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:16:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart
2024-05-08T01:16:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:1 rbd mirror restart
2024-05-08T01:16:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:16:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:16:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:1
2024-05-08T01:16:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:16:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:27.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:16:27.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:27.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:16:27.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:16:27.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:27.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:16:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:16:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:27.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:27.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:27.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:27.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:16:27.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:16:27.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:16:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart
2024-05-08T01:16:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:2 rbd mirror restart
2024-05-08T01:16:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:16:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:16:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:2
2024-05-08T01:16:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:16:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:16:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:16:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:16:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:16:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:16:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:16:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:16:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:16:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror restart
2024-05-08T01:16:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:3 rbd mirror restart
2024-05-08T01:16:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:16:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:16:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:3
2024-05-08T01:16:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:16:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:16:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:16:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:16:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:16:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:16:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:16:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:16:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:16:27.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:27.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror restart
2024-05-08T01:16:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:16:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:16:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:16:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:16:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:16:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:16:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:16:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:16:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:16:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:16:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:16:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:16:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:16:29.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:29.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:16:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:16:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:16:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:16:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:16:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:16:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:16:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:16:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:16:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test1
2024-05-08T01:16:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started
2024-05-08T01:16:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:16:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test1 started
2024-05-08T01:16:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:16:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:16:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:16:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1
2024-05-08T01:16:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:16:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:16:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:16:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:16:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:16:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1
2024-05-08T01:16:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1
2024-05-08T01:16:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:16:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:16:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:16:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:16:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:16:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:16:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:16:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:16:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:16:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:16:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:16:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   757bbca6-4715-40fb-ae9f-f4772f7276b1
2024-05-08T01:16:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:16:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}
2024-05-08T01:16:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:16:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:16:27'
2024-05-08T01:16:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:16:31.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:16:31.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:16:31.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}
2024-05-08T01:16:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:16:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:16:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:16:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:16:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:16:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:16:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:27'
2024-05-08T01:16:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:16:27'
2024-05-08T01:16:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:16:27'
2024-05-08T01:16:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:16:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5},&quot;primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:27</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:27</last_update></image></images></status>'
2024-05-08T01:16:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:16:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:16:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:27'
2024-05-08T01:16:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:16:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:16:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:16:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying
2024-05-08T01:16:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:32.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' ''
2024-05-08T01:16:32.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1
2024-05-08T01:16:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test1:
2024-05-08T01:16:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   4d17c619-1bc7-4c02-b113-0e958e942b07
2024-05-08T01:16:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:16:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:16:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:16:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:16:27'
2024-05-08T01:16:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:16:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:16:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:16:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:16:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:16:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:16:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:16:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:16:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:16:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:16:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:27'
2024-05-08T01:16:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2024-05-08 01:16:27'
2024-05-08T01:16:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:16:27'
2024-05-08T01:16:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:16:32.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5},&quot;primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:27</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:27</last_update></image></images></status>'
2024-05-08T01:16:32.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:16:32.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update'
2024-05-08T01:16:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:27'
2024-05-08T01:16:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state'
2024-05-08T01:16:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:16:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:16:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ all_admin_daemons cluster1 rbd mirror stop
2024-05-08T01:16:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:16:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror stop
2024-05-08T01:16:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:16:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:16:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:16:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:16:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:16:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop
2024-05-08T01:16:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:1 rbd mirror stop
2024-05-08T01:16:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:16:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:16:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:1
2024-05-08T01:16:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:16:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:16:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:16:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:16:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:16:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:16:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:16:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:16:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:16:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop
2024-05-08T01:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:2 rbd mirror stop
2024-05-08T01:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:2
2024-05-08T01:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:16:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:16:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:16:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror stop
2024-05-08T01:16:33.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:33.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:3 rbd mirror stop
2024-05-08T01:16:33.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:16:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:16:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:3
2024-05-08T01:16:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:16:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:16:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:16:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:16:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:16:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:16:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:16:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:16:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:16:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:33.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:33.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:33.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror stop
2024-05-08T01:16:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:16:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:16:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:16:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:16:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:16:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:16:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:16:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:16:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:16:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:16:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:16:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:16:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:16:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:16:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:16:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:16:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:16:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:16:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:16:34.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:16:34.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:34.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1
2024-05-08T01:16:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped
2024-05-08T01:16:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:16:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test1 stopped
2024-05-08T01:16:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:16:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:16:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:16:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1
2024-05-08T01:16:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:16:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:16:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:16:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:16:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:16:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1
2024-05-08T01:16:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1
2024-05-08T01:16:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:16:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:16:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:16:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:16:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:16:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:16:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:16:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:16:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:16:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:16:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:16:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   757bbca6-4715-40fb-ae9f-f4772f7276b1
2024-05-08T01:16:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:16:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:16:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:16:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:16:32'
2024-05-08T01:16:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:16:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:16:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:16:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: stopped
2024-05-08T01:16:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:16:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:16:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:16:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:16:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:16:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:16:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:16:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:16:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:32'
2024-05-08T01:16:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:16:32'
2024-05-08T01:16:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:16:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:16:32'
2024-05-08T01:16:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:16:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+stopped</state><description>stopped</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:32</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+stopped</state><description>stopped</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:32</last_update></image></images></status>'
2024-05-08T01:16:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:16:36.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:16:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:32'
2024-05-08T01:16:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:16:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:16:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:16:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:16:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:16:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped
2024-05-08T01:16:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:16:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' ''
2024-05-08T01:16:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:16:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1
2024-05-08T01:16:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test1:
2024-05-08T01:16:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   4d17c619-1bc7-4c02-b113-0e958e942b07
2024-05-08T01:16:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:16:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:16:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:16:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:16:32'
2024-05-08T01:16:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:16:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:16:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:16:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: stopped
2024-05-08T01:16:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:16:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:16:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:16:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:16:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:16:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:16:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:16:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:16:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:32'
2024-05-08T01:16:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2024-05-08 01:16:32'
2024-05-08T01:16:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:16:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:16:32'
2024-05-08T01:16:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:16:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+stopped</state><description>stopped</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:32</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+stopped</state><description>stopped</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:32</last_update></image></images></status>'
2024-05-08T01:16:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:16:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update'
2024-05-08T01:16:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:32'
2024-05-08T01:16:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state'
2024-05-08T01:16:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:16:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:16:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:16:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:16:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ all_admin_daemons cluster1 rbd mirror restart
2024-05-08T01:16:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:16:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror restart
2024-05-08T01:16:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:16:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:16:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:16:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:16:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:16:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart
2024-05-08T01:16:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:1 rbd mirror restart
2024-05-08T01:16:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:16:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:16:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:1
2024-05-08T01:16:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:16:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:16:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:16:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:16:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:16:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:16:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:16:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:16:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:16:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart
2024-05-08T01:16:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:2 rbd mirror restart
2024-05-08T01:16:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:16:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:16:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:2
2024-05-08T01:16:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:16:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:16:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:16:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:16:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:16:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:16:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:16:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:16:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:16:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror restart
2024-05-08T01:16:38.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:38.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:3 rbd mirror restart
2024-05-08T01:16:38.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:16:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:16:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:3
2024-05-08T01:16:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:16:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:16:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:16:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:16:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:16:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:16:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:16:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:16:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:16:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror restart
2024-05-08T01:16:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:16:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:16:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:16:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:16:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:16:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:16:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:16:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:16:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:16:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:16:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:16:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:16:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:16:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:16:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:16:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:16:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:16:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:16:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:16:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:16:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:16:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:16:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test1
2024-05-08T01:16:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started
2024-05-08T01:16:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:16:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test1 started
2024-05-08T01:16:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:16:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:16:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:16:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1
2024-05-08T01:16:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:16:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:16:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:16:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:16:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:16:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1
2024-05-08T01:16:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:40.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:40.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1
2024-05-08T01:16:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:16:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:16:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:16:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:16:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:16:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:16:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:16:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:16:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:16:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:16:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:16:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   757bbca6-4715-40fb-ae9f-f4772f7276b1
2024-05-08T01:16:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:16:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}
2024-05-08T01:16:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:16:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:16:38'
2024-05-08T01:16:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:16:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:16:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:16:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}
2024-05-08T01:16:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:16:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:16:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:16:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:16:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:16:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:16:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:38'
2024-05-08T01:16:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:16:38'
2024-05-08T01:16:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:41.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:16:38'
2024-05-08T01:16:41.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:41.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:41.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:16:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5},&quot;primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:38</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:38</last_update></image></images></status>'
2024-05-08T01:16:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:16:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:16:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:38'
2024-05-08T01:16:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:16:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:16:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:16:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying
2024-05-08T01:16:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' ''
2024-05-08T01:16:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1
2024-05-08T01:16:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test1:
2024-05-08T01:16:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   4d17c619-1bc7-4c02-b113-0e958e942b07
2024-05-08T01:16:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:16:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:16:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:16:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:16:38'
2024-05-08T01:16:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:16:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:16:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:16:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:16:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:16:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:16:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:16:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:16:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:16:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:16:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:38'
2024-05-08T01:16:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2024-05-08 01:16:38'
2024-05-08T01:16:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:16:38'
2024-05-08T01:16:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:16:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5},&quot;primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:38</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:38</last_update></image></images></status>'
2024-05-08T01:16:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:16:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update'
2024-05-08T01:16:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:38'
2024-05-08T01:16:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state'
2024-05-08T01:16:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:16:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:16:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ all_admin_daemons cluster1 rbd mirror stop mirror cluster2
2024-05-08T01:16:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:16:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror cluster2
2024-05-08T01:16:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:16:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:16:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:16:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:16:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:16:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:43.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror cluster2
2024-05-08T01:16:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:1 rbd mirror stop mirror cluster2
2024-05-08T01:16:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:16:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:16:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:1
2024-05-08T01:16:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:16:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:16:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:16:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:16:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:16:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:16:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:16:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:16:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:16:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop mirror cluster2
2024-05-08T01:16:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:2 rbd mirror stop mirror cluster2
2024-05-08T01:16:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:2
2024-05-08T01:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:16:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:16:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror stop mirror cluster2
2024-05-08T01:16:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:3 rbd mirror stop mirror cluster2
2024-05-08T01:16:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:16:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:16:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:3
2024-05-08T01:16:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:16:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:16:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:16:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:16:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:16:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:16:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:16:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:16:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:16:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror stop mirror cluster2
2024-05-08T01:16:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:16:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:16:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:16:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:16:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:16:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:16:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:16:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:16:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:16:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:16:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:16:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:16:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:16:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:16:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:16:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:16:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:16:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:16:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:16:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:16:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1
2024-05-08T01:16:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped
2024-05-08T01:16:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:16:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test1 stopped
2024-05-08T01:16:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:16:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:16:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:16:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1
2024-05-08T01:16:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:16:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:16:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:16:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:16:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:16:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1
2024-05-08T01:16:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1
2024-05-08T01:16:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:16:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:16:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:16:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:16:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:16:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:16:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:16:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:16:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:16:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:16:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:16:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   757bbca6-4715-40fb-ae9f-f4772f7276b1
2024-05-08T01:16:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:16:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:16:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:16:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:16:43'
2024-05-08T01:16:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:16:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:16:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:16:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: stopped
2024-05-08T01:16:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:16:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:16:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:16:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:16:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:16:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:16:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:16:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:16:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:43'
2024-05-08T01:16:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:16:43'
2024-05-08T01:16:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:16:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:16:43'
2024-05-08T01:16:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:16:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+stopped</state><description>stopped</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:43</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+stopped</state><description>stopped</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:43</last_update></image></images></status>'
2024-05-08T01:16:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:16:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:16:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:43'
2024-05-08T01:16:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:16:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:16:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:16:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:16:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:16:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped
2024-05-08T01:16:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:16:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' ''
2024-05-08T01:16:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1
2024-05-08T01:16:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test1:
2024-05-08T01:16:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   4d17c619-1bc7-4c02-b113-0e958e942b07
2024-05-08T01:16:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:16:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:16:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:16:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:16:43'
2024-05-08T01:16:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:16:48.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:16:48.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:16:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: stopped
2024-05-08T01:16:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:16:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:16:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:16:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:16:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:16:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:16:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:16:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:16:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:43'
2024-05-08T01:16:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2024-05-08 01:16:43'
2024-05-08T01:16:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:16:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:16:43'
2024-05-08T01:16:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:16:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+stopped</state><description>stopped</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:43</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+stopped</state><description>stopped</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:43</last_update></image></images></status>'
2024-05-08T01:16:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:16:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update'
2024-05-08T01:16:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:43'
2024-05-08T01:16:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state'
2024-05-08T01:16:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:16:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:16:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:16:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:16:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror start mirror/test
2024-05-08T01:16:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:16:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:16:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:16:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:16:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:16:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/test
2024-05-08T01:16:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:16:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:16:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:16:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:16:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:16:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/test
2024-05-08T01:16:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:16:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:16:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:16:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:16:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:16:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:16:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:16:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:16:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:16:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:16:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:16:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:16:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:16:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:16:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:16:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:16:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:16:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:16:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:16:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:16:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:16:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:16:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:16:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:16:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:16:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:16:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   757bbca6-4715-40fb-ae9f-f4772f7276b1
2024-05-08T01:16:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:16:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}
2024-05-08T01:16:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:16:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:16:48'
2024-05-08T01:16:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:16:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:16:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:16:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}
2024-05-08T01:16:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:16:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:16:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:16:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:16:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:16:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:16:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:48'
2024-05-08T01:16:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:16:48'
2024-05-08T01:16:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:16:48'
2024-05-08T01:16:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:16:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5},&quot;primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:48</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+stopped</state><description>stopped</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:43</last_update></image></images></status>'
2024-05-08T01:16:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:16:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:16:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:48'
2024-05-08T01:16:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:16:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:16:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:16:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ all_admin_daemons cluster1 rbd mirror start mirror cluster2
2024-05-08T01:16:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:16:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror cluster2
2024-05-08T01:16:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:16:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:16:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:16:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:16:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:16:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror cluster2
2024-05-08T01:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:1 rbd mirror start mirror cluster2
2024-05-08T01:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:1
2024-05-08T01:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:16:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:16:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:16:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:16:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:16:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:16:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:16:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:16:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:16:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror start mirror cluster2
2024-05-08T01:16:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:2 rbd mirror start mirror cluster2
2024-05-08T01:16:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:16:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:16:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:2
2024-05-08T01:16:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:16:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:16:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:16:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:16:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:16:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:16:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:16:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:16:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:16:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror start mirror cluster2
2024-05-08T01:16:51.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:3 rbd mirror start mirror cluster2
2024-05-08T01:16:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:16:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:16:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:3
2024-05-08T01:16:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:16:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:16:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:16:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:16:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:16:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:16:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:16:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:16:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:16:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror start mirror cluster2
2024-05-08T01:16:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test1
2024-05-08T01:16:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started
2024-05-08T01:16:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:16:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test1 started
2024-05-08T01:16:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:16:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:16:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:16:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1
2024-05-08T01:16:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:16:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:16:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:16:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:16:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:16:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1
2024-05-08T01:16:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:52.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:52.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1
2024-05-08T01:16:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:16:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:16:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:16:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:16:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:16:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:16:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:16:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying
2024-05-08T01:16:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' ''
2024-05-08T01:16:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1
2024-05-08T01:16:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test1:
2024-05-08T01:16:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   4d17c619-1bc7-4c02-b113-0e958e942b07
2024-05-08T01:16:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:16:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:16:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:16:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:16:51'
2024-05-08T01:16:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:16:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:16:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:16:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:16:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:16:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:16:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:16:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:16:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:16:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:16:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:51'
2024-05-08T01:16:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2024-05-08 01:16:51'
2024-05-08T01:16:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:16:51'
2024-05-08T01:16:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:16:53.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5},&quot;primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:48</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:51</last_update></image></images></status>'
2024-05-08T01:16:53.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:16:53.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update'
2024-05-08T01:16:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:51'
2024-05-08T01:16:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state'
2024-05-08T01:16:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:16:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:16:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror restart mirror/test
2024-05-08T01:16:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:16:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:16:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:16:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:16:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:16:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror/test
2024-05-08T01:16:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:16:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:16:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:16:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:16:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:16:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror/test
2024-05-08T01:16:54.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:54.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:16:54.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:54.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:54.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:16:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:16:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:16:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:16:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:16:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:16:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:16:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:16:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:16:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:16:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:16:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:16:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:16:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:16:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:16:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:16:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:16:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:16:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:16:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:16:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:16:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:16:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:16:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:16:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:16:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:16:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:16:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   757bbca6-4715-40fb-ae9f-f4772f7276b1
2024-05-08T01:16:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:16:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}
2024-05-08T01:16:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:16:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:16:54'
2024-05-08T01:16:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:16:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:16:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:16:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}
2024-05-08T01:16:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:16:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:16:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:16:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:16:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:16:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:16:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:54'
2024-05-08T01:16:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:16:54'
2024-05-08T01:16:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:16:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:16:54'
2024-05-08T01:16:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:16:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:16:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:16:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5},&quot;primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:54</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:51</last_update></image></images></status>'
2024-05-08T01:16:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:16:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:16:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:54'
2024-05-08T01:16:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:16:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:16:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:16:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:16:56.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:16:56.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:56.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:56.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ all_admin_daemons cluster1 rbd mirror restart mirror cluster2
2024-05-08T01:16:56.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:56.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:56.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:16:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror cluster2
2024-05-08T01:16:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:16:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:56.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:16:56.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:56.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:56.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:56.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:16:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:16:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:16:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror cluster2
2024-05-08T01:16:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:1 rbd mirror restart mirror cluster2
2024-05-08T01:16:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:16:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:16:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:1
2024-05-08T01:16:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:16:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:16:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:16:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:16:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:16:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:16:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:16:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:16:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:16:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart mirror cluster2
2024-05-08T01:16:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:2 rbd mirror restart mirror cluster2
2024-05-08T01:16:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:16:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:16:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:2
2024-05-08T01:16:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:16:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:16:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:16:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:16:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:16:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:16:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:16:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:16:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:16:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror restart mirror cluster2
2024-05-08T01:16:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:3 rbd mirror restart mirror cluster2
2024-05-08T01:16:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:16:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:16:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:3
2024-05-08T01:16:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:16:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:16:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:16:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:16:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:16:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:16:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:16:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:16:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:16:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror restart mirror cluster2
2024-05-08T01:16:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:16:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:16:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:16:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:16:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:16:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:16:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:16:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:16:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:16:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:16:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:16:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:16:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:16:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:16:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:16:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:16:58.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:16:58.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:16:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:16:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:16:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:16:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:16:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test1
2024-05-08T01:16:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started
2024-05-08T01:16:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:16:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:16:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test1 started
2024-05-08T01:16:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:16:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:16:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:16:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:16:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1
2024-05-08T01:16:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:16:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:16:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:16:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:16:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:16:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:16:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:16:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1
2024-05-08T01:16:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:16:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:16:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:16:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:16:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:16:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:16:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:16:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:16:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:16:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:16:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:16:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:16:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:16:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:16:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:16:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1
2024-05-08T01:16:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:16:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:16:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:16:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:16:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:16:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:16:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:16:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:16:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ all_admin_daemons cluster1 rbd mirror stop mirror cluster2
2024-05-08T01:16:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:16:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror cluster2
2024-05-08T01:16:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:16:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:16:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:16:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:16:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:16:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:16:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:16:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:16:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:16:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:16:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:16:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:16:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:16:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:16:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:16:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror cluster2
2024-05-08T01:16:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:1 rbd mirror stop mirror cluster2
2024-05-08T01:16:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:16:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:16:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:1
2024-05-08T01:16:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:16:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:16:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:16:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:16:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:16:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:16:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:16:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:16:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:16:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:16:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:16:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:16:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop mirror cluster2
2024-05-08T01:16:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:2 rbd mirror stop mirror cluster2
2024-05-08T01:16:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:16:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:16:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:2
2024-05-08T01:16:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:16:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:16:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:16:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:16:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:16:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror stop mirror cluster2
2024-05-08T01:16:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:16:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:3 rbd mirror stop mirror cluster2
2024-05-08T01:16:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:16:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:16:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:16:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:16:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:3
2024-05-08T01:16:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:16:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:16:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:16:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:16:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:16:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:16:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:16:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:16:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:16:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:16:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:16:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:16:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:16:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:16:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:16:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:16:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:16:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:16:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:16:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:16:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:16:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:16:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:16:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror stop mirror cluster2
2024-05-08T01:16:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:16:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:16:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:16:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:16:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:16:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:16:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:16:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:16:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:17:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:17:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:17:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:17:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:17:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:17:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:17:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:17:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:17:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:17:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:17:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:17:01.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:17:01.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:17:01.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:17:01.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:17:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:17:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1
2024-05-08T01:17:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:17:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped
2024-05-08T01:17:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:17:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:17:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:17:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test1 stopped
2024-05-08T01:17:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:17:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:17:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:17:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:17:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1
2024-05-08T01:17:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:17:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:17:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:17:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:17:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:17:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1
2024-05-08T01:17:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:02.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:02.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:02.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:02.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1
2024-05-08T01:17:02.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:17:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:17:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:17:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:17:02.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:17:02.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:02.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:17:02.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:17:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:17:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:17:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:17:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:17:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   757bbca6-4715-40fb-ae9f-f4772f7276b1
2024-05-08T01:17:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:17:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:17:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:17:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:16:59'
2024-05-08T01:17:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:17:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:17:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:17:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: stopped
2024-05-08T01:17:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:17:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:17:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:17:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:17:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:17:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:17:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:17:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:17:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:59'
2024-05-08T01:17:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:16:59'
2024-05-08T01:17:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:17:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:16:59'
2024-05-08T01:17:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:17:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+stopped</state><description>stopped</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:59</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+stopped</state><description>stopped</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:59</last_update></image></images></status>'
2024-05-08T01:17:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:17:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:17:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:59'
2024-05-08T01:17:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:17:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:17:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:17:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:17:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:17:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped
2024-05-08T01:17:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:17:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:17:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' ''
2024-05-08T01:17:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:17:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:17:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1
2024-05-08T01:17:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test1:
2024-05-08T01:17:04.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   4d17c619-1bc7-4c02-b113-0e958e942b07
2024-05-08T01:17:04.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:17:04.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:17:04.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:17:04.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:16:59'
2024-05-08T01:17:04.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:17:04.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:17:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:17:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: stopped
2024-05-08T01:17:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:17:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:17:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:17:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:17:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:17:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:17:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:17:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:17:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:59'
2024-05-08T01:17:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2024-05-08 01:16:59'
2024-05-08T01:17:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:17:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:17:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:16:59'
2024-05-08T01:17:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:17:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+stopped</state><description>stopped</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:59</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+stopped</state><description>stopped</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:16:59</last_update></image></images></status>'
2024-05-08T01:17:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:17:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update'
2024-05-08T01:17:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:16:59'
2024-05-08T01:17:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state'
2024-05-08T01:17:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:17:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:17:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:17:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:17:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ all_admin_daemons cluster1 rbd mirror restart mirror cluster2
2024-05-08T01:17:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:17:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror cluster2
2024-05-08T01:17:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:17:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:04.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:04.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:04.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:17:04.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:04.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:17:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:17:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:17:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:17:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:17:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:17:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:17:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:17:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:04.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror cluster2
2024-05-08T01:17:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:1 rbd mirror restart mirror cluster2
2024-05-08T01:17:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:17:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:17:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:1
2024-05-08T01:17:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:17:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:17:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster1:1 ']'
2024-05-08T01:17:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:17:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:17:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:17:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:17:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:17:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:17:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:17:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:17:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:17:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:17:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:17:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:17:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:17:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:17:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:17:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart mirror cluster2
2024-05-08T01:17:04.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:04.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:2 rbd mirror restart mirror cluster2
2024-05-08T01:17:04.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:17:04.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:17:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:2
2024-05-08T01:17:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:17:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:17:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster1:2 ']'
2024-05-08T01:17:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:17:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:17:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:17:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:17:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:17:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:17:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:17:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:17:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:17:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:17:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:17:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:17:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:17:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:17:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:17:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror restart mirror cluster2
2024-05-08T01:17:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:3 rbd mirror restart mirror cluster2
2024-05-08T01:17:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:17:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:17:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:3
2024-05-08T01:17:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:17:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:17:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster1:3 ']'
2024-05-08T01:17:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:17:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:17:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:17:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:17:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:17:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:17:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:17:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:17:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:17:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:17:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:17:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:17:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:17:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:17:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:17:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror restart mirror cluster2
2024-05-08T01:17:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:17:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:17:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:17:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:17:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:17:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:17:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:17:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:17:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:17:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:17:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:17:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:17:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:17:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:17:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:17:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:06.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:06.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:06.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:06.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:06.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:06.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:06.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:06.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:06.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:06.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:06.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:06.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:06.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:06.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:06.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:06.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:06.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:17:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:17:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:17:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:17:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:17:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:17:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:17:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:17:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test1
2024-05-08T01:17:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:17:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started
2024-05-08T01:17:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:17:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:17:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:17:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test1 started
2024-05-08T01:17:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:17:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:17:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:17:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:17:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1
2024-05-08T01:17:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:17:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:17:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:17:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:17:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:17:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1
2024-05-08T01:17:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1
2024-05-08T01:17:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:17:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:17:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:17:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:17:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:17:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:17:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:17:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:17:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:17:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:17:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:17:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:08.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:08.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:17:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:17:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   757bbca6-4715-40fb-ae9f-f4772f7276b1
2024-05-08T01:17:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:17:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}
2024-05-08T01:17:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:17:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:17:04'
2024-05-08T01:17:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:17:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:17:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:17:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}
2024-05-08T01:17:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:17:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:17:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:17:08.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:17:08.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:17:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:17:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:17:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:17:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:04'
2024-05-08T01:17:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:17:04'
2024-05-08T01:17:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:17:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:17:04'
2024-05-08T01:17:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:17:08.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5},&quot;primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:17:04</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:17:04</last_update></image></images></status>'
2024-05-08T01:17:08.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:17:08.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:17:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:04'
2024-05-08T01:17:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:17:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:17:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:17:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:17:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:17:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying
2024-05-08T01:17:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:17:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:17:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' ''
2024-05-08T01:17:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:17:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:17:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1
2024-05-08T01:17:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test1:
2024-05-08T01:17:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   4d17c619-1bc7-4c02-b113-0e958e942b07
2024-05-08T01:17:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:17:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:17:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:17:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:17:04'
2024-05-08T01:17:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:17:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:17:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:17:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:17:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:17:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:17:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:17:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:17:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:17:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:17:09.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:17:09.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:17:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:04'
2024-05-08T01:17:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2024-05-08 01:17:04'
2024-05-08T01:17:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:17:09.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:17:09.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:17:04'
2024-05-08T01:17:09.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:09.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:09.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:17:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>2</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5},&quot;primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:17:04</last_update></image><image><name>test1</name><global_id>4d17c619-1bc7-4c02-b113-0e958e942b07</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:17:04</last_update></image></images></status>'
2024-05-08T01:17:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:17:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update'
2024-05-08T01:17:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:04'
2024-05-08T01:17:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state'
2024-05-08T01:17:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:17:09.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:17:09.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:17:09.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:17:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1
2024-05-08T01:17:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=
2024-05-08T01:17:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=
2024-05-08T01:17:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:17:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:17:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush
2024-05-08T01:17:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:17:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:17:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:17:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:17:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:17:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush
2024-05-08T01:17:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:17:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:17:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:17:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:17:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:17:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:17:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:17:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:17:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:17:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:17:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush
2024-05-08T01:17:09.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:09.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ all_admin_daemons cluster1 rbd mirror status
2024-05-08T01:17:09.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:17:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror status
2024-05-08T01:17:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:17:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:17:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:17:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:17:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:17:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:17:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:09.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:17:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:17:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:17:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:17:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status
2024-05-08T01:17:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:{
2024-05-08T01:17:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "pool_replayers": [
2024-05-08T01:17:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:        {
2024-05-08T01:17:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "peer": "uuid: 8fda577f-a20d-47c9-89a8-9e056ea57c49 cluster: cluster2 client: client.mirror",
2024-05-08T01:17:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "pool": "mirror",
2024-05-08T01:17:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "instance_id": "4318",
2024-05-08T01:17:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "state": "stopped (manual)",
2024-05-08T01:17:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "leader_instance_id": "4318",
2024-05-08T01:17:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "leader": true,
2024-05-08T01:17:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "instances": [
2024-05-08T01:17:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "4318",
2024-05-08T01:17:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "4323",
2024-05-08T01:17:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "4327",
2024-05-08T01:17:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "4329"
2024-05-08T01:17:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            ],
2024-05-08T01:17:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "local_cluster_admin_socket": "/var/run/ceph/client.mirror.0.31313.cluster1.94919394203888.asok",
2024-05-08T01:17:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.31313.cluster2.94919406868720.asok",
2024-05-08T01:17:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "sync_throttler": {
2024-05-08T01:17:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "max_parallel_requests": 5,
2024-05-08T01:17:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "running_requests": 0,
2024-05-08T01:17:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "waiting_requests": 0
2024-05-08T01:17:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            },
2024-05-08T01:17:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "deletion_throttler": {
2024-05-08T01:17:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "max_parallel_requests": 1,
2024-05-08T01:17:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "running_requests": 0,
2024-05-08T01:17:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "waiting_requests": 0
2024-05-08T01:17:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            },
2024-05-08T01:17:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "image_replayers": [
2024-05-08T01:17:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                {
2024-05-08T01:17:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "name": "mirror/test1",
2024-05-08T01:17:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "state": "Replaying"
2024-05-08T01:17:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                },
2024-05-08T01:17:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                {
2024-05-08T01:17:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "name": "mirror/test",
2024-05-08T01:17:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "state": "Replaying"
2024-05-08T01:17:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                }
2024-05-08T01:17:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            ],
2024-05-08T01:17:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "image_deleter": {
2024-05-08T01:17:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "image_deleter_status": {
2024-05-08T01:17:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "delete_images_queue": [],
2024-05-08T01:17:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "failed_deletes_queue": []
2024-05-08T01:17:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                }
2024-05-08T01:17:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            },
2024-05-08T01:17:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "namespaces": [
2024-05-08T01:17:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                {
2024-05-08T01:17:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "name": "ns1",
2024-05-08T01:17:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "image_replayers": [],
2024-05-08T01:17:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "image_deleter": {
2024-05-08T01:17:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                        "image_deleter_status": {
2024-05-08T01:17:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                            "delete_images_queue": [],
2024-05-08T01:17:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                            "failed_deletes_queue": []
2024-05-08T01:17:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                        }
2024-05-08T01:17:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    }
2024-05-08T01:17:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                },
2024-05-08T01:17:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                {
2024-05-08T01:17:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "name": "ns2",
2024-05-08T01:17:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "image_replayers": [],
2024-05-08T01:17:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "image_deleter": {
2024-05-08T01:17:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                        "image_deleter_status": {
2024-05-08T01:17:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                            "delete_images_queue": [],
2024-05-08T01:17:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                            "failed_deletes_queue": []
2024-05-08T01:17:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                        }
2024-05-08T01:17:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    }
2024-05-08T01:17:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                }
2024-05-08T01:17:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            ]
2024-05-08T01:17:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:        },
2024-05-08T01:17:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:        {
2024-05-08T01:17:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "peer": "uuid: d67a7ec3-c140-49f2-84db-ae12a16e75f9 cluster: cluster2 client: client.mirror",
2024-05-08T01:17:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "pool": "mirror_parent",
2024-05-08T01:17:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "instance_id": "4332",
2024-05-08T01:17:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "state": "stopped (manual)",
2024-05-08T01:17:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "leader_instance_id": "4332",
2024-05-08T01:17:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "leader": true,
2024-05-08T01:17:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "instances": [
2024-05-08T01:17:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "4332",
2024-05-08T01:17:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "4334",
2024-05-08T01:17:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "4336",
2024-05-08T01:17:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "4338"
2024-05-08T01:17:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            ],
2024-05-08T01:17:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "local_cluster_admin_socket": "/var/run/ceph/client.mirror.0.31313.cluster1.94919394564336.asok",
2024-05-08T01:17:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.31313.cluster2.94919396489456.asok",
2024-05-08T01:17:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "sync_throttler": {
2024-05-08T01:17:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "max_parallel_requests": 5,
2024-05-08T01:17:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "running_requests": 0,
2024-05-08T01:17:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "waiting_requests": 0
2024-05-08T01:17:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            },
2024-05-08T01:17:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "deletion_throttler": {
2024-05-08T01:17:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "max_parallel_requests": 1,
2024-05-08T01:17:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "running_requests": 0,
2024-05-08T01:17:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "waiting_requests": 0
2024-05-08T01:17:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            },
2024-05-08T01:17:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "image_replayers": [],
2024-05-08T01:17:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "image_deleter": {
2024-05-08T01:17:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "image_deleter_status": {
2024-05-08T01:17:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "delete_images_queue": [],
2024-05-08T01:17:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "failed_deletes_queue": []
2024-05-08T01:17:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                }
2024-05-08T01:17:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            },
2024-05-08T01:17:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "namespaces": []
2024-05-08T01:17:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:        }
2024-05-08T01:17:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    ]
2024-05-08T01:17:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:}
2024-05-08T01:17:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:1 rbd mirror status
2024-05-08T01:17:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:17:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:17:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:1
2024-05-08T01:17:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:17:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:17:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster1:1 ']'
2024-05-08T01:17:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:17:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:17:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:17:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:17:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:17:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:17:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:17:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:17:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:17:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:17:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:17:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:17:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:17:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:17:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:17:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status
2024-05-08T01:17:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:{
2024-05-08T01:17:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "pool_replayers": [
2024-05-08T01:17:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:        {
2024-05-08T01:17:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "peer": "uuid: 8fda577f-a20d-47c9-89a8-9e056ea57c49 cluster: cluster2 client: client.mirror",
2024-05-08T01:17:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "pool": "mirror",
2024-05-08T01:17:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "instance_id": "4323",
2024-05-08T01:17:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "state": "stopped (manual)",
2024-05-08T01:17:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "leader_instance_id": "4318",
2024-05-08T01:17:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "leader": false,
2024-05-08T01:17:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "local_cluster_admin_socket": "/var/run/ceph/client.mirror.1.31341.cluster1.94738496551152.asok",
2024-05-08T01:17:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.31341.cluster2.94738509158640.asok",
2024-05-08T01:17:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "sync_throttler": {
2024-05-08T01:17:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "max_parallel_requests": 5,
2024-05-08T01:17:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "running_requests": 0,
2024-05-08T01:17:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "waiting_requests": 0
2024-05-08T01:17:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            },
2024-05-08T01:17:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "deletion_throttler": {
2024-05-08T01:17:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "max_parallel_requests": 1,
2024-05-08T01:17:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "running_requests": 0,
2024-05-08T01:17:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "waiting_requests": 0
2024-05-08T01:17:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            },
2024-05-08T01:17:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "image_replayers": [],
2024-05-08T01:17:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "namespaces": [
2024-05-08T01:17:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                {
2024-05-08T01:17:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "name": "ns1",
2024-05-08T01:17:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "image_replayers": []
2024-05-08T01:17:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                },
2024-05-08T01:17:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                {
2024-05-08T01:17:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "name": "ns2",
2024-05-08T01:17:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "image_replayers": []
2024-05-08T01:17:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                }
2024-05-08T01:17:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            ]
2024-05-08T01:17:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:        },
2024-05-08T01:17:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:        {
2024-05-08T01:17:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "peer": "uuid: d67a7ec3-c140-49f2-84db-ae12a16e75f9 cluster: cluster2 client: client.mirror",
2024-05-08T01:17:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "pool": "mirror_parent",
2024-05-08T01:17:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "instance_id": "4334",
2024-05-08T01:17:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "state": "stopped (manual)",
2024-05-08T01:17:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "leader_instance_id": "4332",
2024-05-08T01:17:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "leader": false,
2024-05-08T01:17:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "local_cluster_admin_socket": "/var/run/ceph/client.mirror.1.31341.cluster1.94738496936176.asok",
2024-05-08T01:17:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.31341.cluster2.94738498902256.asok",
2024-05-08T01:17:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "sync_throttler": {
2024-05-08T01:17:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "max_parallel_requests": 5,
2024-05-08T01:17:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "running_requests": 0,
2024-05-08T01:17:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "waiting_requests": 0
2024-05-08T01:17:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            },
2024-05-08T01:17:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "deletion_throttler": {
2024-05-08T01:17:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "max_parallel_requests": 1,
2024-05-08T01:17:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "running_requests": 0,
2024-05-08T01:17:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "waiting_requests": 0
2024-05-08T01:17:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            },
2024-05-08T01:17:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "image_replayers": [],
2024-05-08T01:17:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "namespaces": []
2024-05-08T01:17:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:        }
2024-05-08T01:17:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    ]
2024-05-08T01:17:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:}
2024-05-08T01:17:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:2 rbd mirror status
2024-05-08T01:17:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:17:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:17:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:2
2024-05-08T01:17:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:17:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:17:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster1:2 ']'
2024-05-08T01:17:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:17:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:17:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:17:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:17:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:17:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:17:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:17:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:17:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:17:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:17:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:17:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:17:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:17:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:17:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:17:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status
2024-05-08T01:17:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:{
2024-05-08T01:17:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "pool_replayers": [
2024-05-08T01:17:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:        {
2024-05-08T01:17:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "peer": "uuid: 8fda577f-a20d-47c9-89a8-9e056ea57c49 cluster: cluster2 client: client.mirror",
2024-05-08T01:17:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "pool": "mirror",
2024-05-08T01:17:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "instance_id": "4327",
2024-05-08T01:17:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "state": "stopped (manual)",
2024-05-08T01:17:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "leader_instance_id": "4318",
2024-05-08T01:17:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "leader": false,
2024-05-08T01:17:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "local_cluster_admin_socket": "/var/run/ceph/client.mirror.2.31369.cluster1.94641465817328.asok",
2024-05-08T01:17:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.31369.cluster2.94641478490352.asok",
2024-05-08T01:17:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "sync_throttler": {
2024-05-08T01:17:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "max_parallel_requests": 5,
2024-05-08T01:17:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "running_requests": 0,
2024-05-08T01:17:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "waiting_requests": 0
2024-05-08T01:17:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            },
2024-05-08T01:17:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "deletion_throttler": {
2024-05-08T01:17:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "max_parallel_requests": 1,
2024-05-08T01:17:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "running_requests": 0,
2024-05-08T01:17:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "waiting_requests": 0
2024-05-08T01:17:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            },
2024-05-08T01:17:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "image_replayers": [],
2024-05-08T01:17:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "namespaces": [
2024-05-08T01:17:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                {
2024-05-08T01:17:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "name": "ns1",
2024-05-08T01:17:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "image_replayers": []
2024-05-08T01:17:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                },
2024-05-08T01:17:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                {
2024-05-08T01:17:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "name": "ns2",
2024-05-08T01:17:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "image_replayers": []
2024-05-08T01:17:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                }
2024-05-08T01:17:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            ]
2024-05-08T01:17:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:        },
2024-05-08T01:17:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:        {
2024-05-08T01:17:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "peer": "uuid: d67a7ec3-c140-49f2-84db-ae12a16e75f9 cluster: cluster2 client: client.mirror",
2024-05-08T01:17:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "pool": "mirror_parent",
2024-05-08T01:17:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "instance_id": "4336",
2024-05-08T01:17:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "state": "stopped (manual)",
2024-05-08T01:17:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "leader_instance_id": "4332",
2024-05-08T01:17:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "leader": false,
2024-05-08T01:17:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "local_cluster_admin_socket": "/var/run/ceph/client.mirror.2.31369.cluster1.94641466317040.asok",
2024-05-08T01:17:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.31369.cluster2.94641468201200.asok",
2024-05-08T01:17:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "sync_throttler": {
2024-05-08T01:17:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "max_parallel_requests": 5,
2024-05-08T01:17:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "running_requests": 0,
2024-05-08T01:17:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "waiting_requests": 0
2024-05-08T01:17:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            },
2024-05-08T01:17:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "deletion_throttler": {
2024-05-08T01:17:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "max_parallel_requests": 1,
2024-05-08T01:17:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "running_requests": 0,
2024-05-08T01:17:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "waiting_requests": 0
2024-05-08T01:17:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            },
2024-05-08T01:17:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "image_replayers": [],
2024-05-08T01:17:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "namespaces": []
2024-05-08T01:17:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:        }
2024-05-08T01:17:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    ]
2024-05-08T01:17:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:}
2024-05-08T01:17:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:3 rbd mirror status
2024-05-08T01:17:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:17:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:17:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:3
2024-05-08T01:17:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:17:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:17:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster1:3 ']'
2024-05-08T01:17:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:17:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:17:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:17:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:17:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:17:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:17:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:17:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status
2024-05-08T01:17:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:{
2024-05-08T01:17:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "pool_replayers": [
2024-05-08T01:17:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:        {
2024-05-08T01:17:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "peer": "uuid: 8fda577f-a20d-47c9-89a8-9e056ea57c49 cluster: cluster2 client: client.mirror",
2024-05-08T01:17:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "pool": "mirror",
2024-05-08T01:17:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "instance_id": "4329",
2024-05-08T01:17:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "state": "stopped (manual)",
2024-05-08T01:17:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "leader_instance_id": "4318",
2024-05-08T01:17:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "leader": false,
2024-05-08T01:17:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "local_cluster_admin_socket": "/var/run/ceph/client.mirror.3.31439.cluster1.94543254149360.asok",
2024-05-08T01:17:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.31439.cluster2.94543266855152.asok",
2024-05-08T01:17:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "sync_throttler": {
2024-05-08T01:17:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "max_parallel_requests": 5,
2024-05-08T01:17:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "running_requests": 0,
2024-05-08T01:17:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "waiting_requests": 0
2024-05-08T01:17:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            },
2024-05-08T01:17:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "deletion_throttler": {
2024-05-08T01:17:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "max_parallel_requests": 1,
2024-05-08T01:17:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "running_requests": 0,
2024-05-08T01:17:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "waiting_requests": 0
2024-05-08T01:17:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            },
2024-05-08T01:17:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "image_replayers": [],
2024-05-08T01:17:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "namespaces": [
2024-05-08T01:17:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                {
2024-05-08T01:17:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "name": "ns1",
2024-05-08T01:17:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "image_replayers": []
2024-05-08T01:17:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                },
2024-05-08T01:17:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                {
2024-05-08T01:17:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "name": "ns2",
2024-05-08T01:17:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                    "image_replayers": []
2024-05-08T01:17:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                }
2024-05-08T01:17:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            ]
2024-05-08T01:17:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:        },
2024-05-08T01:17:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:        {
2024-05-08T01:17:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "peer": "uuid: d67a7ec3-c140-49f2-84db-ae12a16e75f9 cluster: cluster2 client: client.mirror",
2024-05-08T01:17:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "pool": "mirror_parent",
2024-05-08T01:17:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "instance_id": "4338",
2024-05-08T01:17:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "state": "stopped (manual)",
2024-05-08T01:17:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "leader_instance_id": "4332",
2024-05-08T01:17:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "leader": false,
2024-05-08T01:17:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "local_cluster_admin_socket": "/var/run/ceph/client.mirror.3.31439.cluster1.94543254591728.asok",
2024-05-08T01:17:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.31439.cluster2.94543255853296.asok",
2024-05-08T01:17:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "sync_throttler": {
2024-05-08T01:17:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "max_parallel_requests": 5,
2024-05-08T01:17:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "running_requests": 0,
2024-05-08T01:17:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "waiting_requests": 0
2024-05-08T01:17:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            },
2024-05-08T01:17:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "deletion_throttler": {
2024-05-08T01:17:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "max_parallel_requests": 1,
2024-05-08T01:17:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "running_requests": 0,
2024-05-08T01:17:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:                "waiting_requests": 0
2024-05-08T01:17:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            },
2024-05-08T01:17:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "image_replayers": [],
2024-05-08T01:17:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:            "namespaces": []
2024-05-08T01:17:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:        }
2024-05-08T01:17:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    ]
2024-05-08T01:17:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:}
2024-05-08T01:17:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster2 mirror test1
2024-05-08T01:17:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:17:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:17:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:17:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster2 mirror test1
2024-05-08T01:17:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test1
2024-05-08T01:17:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 snap purge mirror/test1
2024-05-08T01:17:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rm mirror/test1
2024-05-08T01:17:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:17:10.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: test image rename'
2024-05-08T01:17:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:17:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:17:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:17:10 TEST: test image rename
2024-05-08T01:17:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:17:10 TEST: test image rename
2024-05-08T01:17:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ new_name=test_RENAMED
2024-05-08T01:17:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_image cluster2 mirror test test_RENAMED
2024-05-08T01:17:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_name=test_RENAMED
2024-05-08T01:17:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rename mirror/test mirror/test_RENAMED
2024-05-08T01:17:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = snapshot ']'
2024-05-08T01:17:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test_RENAMED
2024-05-08T01:17:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_RENAMED
2024-05-08T01:17:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test_RENAMED started
2024-05-08T01:17:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_RENAMED
2024-05-08T01:17:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:17:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:17:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test_RENAMED started
2024-05-08T01:17:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_RENAMED
2024-05-08T01:17:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:17:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:17:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:17:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_RENAMED
2024-05-08T01:17:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:17:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:17:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:17:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:17:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:17:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED
2024-05-08T01:17:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED
2024-05-08T01:17:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED']
2024-05-08T01:17:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_RENAMED
2024-05-08T01:17:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:17:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:17:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:17:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:17:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:17:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:17:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:17:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:17:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:17:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:17:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:17:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:17:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:17:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:17:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:17:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:17:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_RENAMED
2024-05-08T01:17:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED']
2024-05-08T01:17:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_RENAMED
2024-05-08T01:17:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:17:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:17:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:17:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:17:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:17:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:17:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:17:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:17:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:17:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:17:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:17:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:17:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:17:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:17:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:17:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:17:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_RENAMED
2024-05-08T01:17:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED']
2024-05-08T01:17:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_RENAMED
2024-05-08T01:17:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:17:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:17:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:17:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:17:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:17:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:17:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:17:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:17:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:17:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:17:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:17:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:17:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:17:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:17:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:17:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:17:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_RENAMED
2024-05-08T01:17:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED']
2024-05-08T01:17:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 1
2024-05-08T01:17:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED
2024-05-08T01:17:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED
2024-05-08T01:17:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:17:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:17:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:17:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:17:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:17:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:17:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:17:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying
2024-05-08T01:17:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_RENAMED
2024-05-08T01:17:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:17:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying '' ''
2024-05-08T01:17:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_RENAMED
2024-05-08T01:17:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:17:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_RENAMED
2024-05-08T01:17:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test_RENAMED:
2024-05-08T01:17:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   757bbca6-4715-40fb-ae9f-f4772f7276b1
2024-05-08T01:17:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:17:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}
2024-05-08T01:17:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:17:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:17:04'
2024-05-08T01:17:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:17:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:17:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:17:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}
2024-05-08T01:17:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:17:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:17:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:17:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:17:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:17:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:17:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:17:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:17:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:04'
2024-05-08T01:17:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_RENAMED up+replaying '2024-05-08 01:17:04'
2024-05-08T01:17:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_RENAMED
2024-05-08T01:17:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:17:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:17:04'
2024-05-08T01:17:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:17:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test_RENAMED</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5},&quot;primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:5}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:17:04</last_update></image></images></status>'
2024-05-08T01:17:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:17:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/last_update'
2024-05-08T01:17:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:04'
2024-05-08T01:17:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/state'
2024-05-08T01:17:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:17:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:17:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:17:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:17:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test_RENAMED
2024-05-08T01:17:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:17:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:17:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:17:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:17:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:17:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED
2024-05-08T01:17:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:17:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:17:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:17:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:17:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:17:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:17:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:17:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:17:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:17:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:17:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED
2024-05-08T01:17:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:{
2024-05-08T01:17:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "name": "mirror/test_RENAMED",
2024-05-08T01:17:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:17:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:}
2024-05-08T01:17:14.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:14.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror restart mirror/test_RENAMED
2024-05-08T01:17:14.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:17:14.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:14.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:14.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:17:14.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:17:14.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:14.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:17:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:17:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror/test_RENAMED
2024-05-08T01:17:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:17:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:17:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:17:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:17:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:17:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:17:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:17:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:17:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:17:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:17:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror/test_RENAMED
2024-05-08T01:17:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test_RENAMED
2024-05-08T01:17:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_RENAMED
2024-05-08T01:17:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test_RENAMED started
2024-05-08T01:17:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_RENAMED
2024-05-08T01:17:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:17:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:17:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test_RENAMED started
2024-05-08T01:17:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_RENAMED
2024-05-08T01:17:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:17:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:17:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:17:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_RENAMED
2024-05-08T01:17:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:17:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:17:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:17:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:17:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:17:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED
2024-05-08T01:17:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED
2024-05-08T01:17:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:17:16.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:17:16.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:17:16.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:17:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:17:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:17:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:17:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying
2024-05-08T01:17:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_RENAMED
2024-05-08T01:17:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:17:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying '' ''
2024-05-08T01:17:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_RENAMED
2024-05-08T01:17:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:17:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_RENAMED
2024-05-08T01:17:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test_RENAMED:
2024-05-08T01:17:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   757bbca6-4715-40fb-ae9f-f4772f7276b1
2024-05-08T01:17:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:17:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6}}
2024-05-08T01:17:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:17:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:17:15'
2024-05-08T01:17:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:17:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:17:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:17:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6}}
2024-05-08T01:17:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:17:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:17:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:17:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:17:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:17:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:17:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:17:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:17:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:15'
2024-05-08T01:17:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_RENAMED up+replaying '2024-05-08 01:17:15'
2024-05-08T01:17:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_RENAMED
2024-05-08T01:17:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:17:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:17:15'
2024-05-08T01:17:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:17:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test_RENAMED</name><global_id>757bbca6-4715-40fb-ae9f-f4772f7276b1</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:1,&quot;object_number&quot;:1,&quot;tag_tid&quot;:6},&quot;primary_position&quot;:{&quot;entry_tid&quot;:1,&quot;object_number&quot;:1,&quot;tag_tid&quot;:6}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:17:15</last_update></image></images></status>'
2024-05-08T01:17:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:17:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/last_update'
2024-05-08T01:17:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:15'
2024-05-08T01:17:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/state'
2024-05-08T01:17:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:17:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:17:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:17:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:17:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_image cluster2 mirror test_RENAMED test
2024-05-08T01:17:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_RENAMED
2024-05-08T01:17:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_name=test
2024-05-08T01:17:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rename mirror/test_RENAMED mirror/test
2024-05-08T01:17:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = snapshot ']'
2024-05-08T01:17:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:17:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:17:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:17:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:17:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:17:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:17:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:17:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:17:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:17:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:17:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:17:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:17:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:17:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:17:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:17:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:18.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:18.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:18.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:18.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:17:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:no valid command found; 1 closest matches:
2024-05-08T01:17:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status mirror/test_RENAMED
2024-05-08T01:17:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test
2024-05-08T01:17:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:17:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:17:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:17:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:17:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:17:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:17:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:17:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:17:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:17:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:17:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:17:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:17:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:17:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:17:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:17:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:17:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test
2024-05-08T01:17:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:17:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test
2024-05-08T01:17:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:17:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:17:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:17:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:17:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:17:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:17:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:17:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:17:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:17:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:17:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:17:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:17:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:17:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:17:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:17:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:17:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test
2024-05-08T01:17:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:17:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test
2024-05-08T01:17:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:17:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:17:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:17:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:17:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:17:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:17:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:17:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:17:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:17:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:17:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:17:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:17:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:17:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:17:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:17:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:17:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test
2024-05-08T01:17:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:17:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 1
2024-05-08T01:17:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:17:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:17:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:17:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:17:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:17:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:17:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:17:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:17:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:17:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: test trash move restore'
2024-05-08T01:17:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:17:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:17:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:17:20 TEST: test trash move restore
2024-05-08T01:17:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:17:20 TEST: test trash move restore
2024-05-08T01:17:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster2 mirror test
2024-05-08T01:17:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster2 info mirror/test
2024-05-08T01:17:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:17:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=10aa2fb4425e
2024-05-08T01:17:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ trash_move cluster2 mirror test
2024-05-08T01:17:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 trash move mirror/test
2024-05-08T01:17:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test deleted
2024-05-08T01:17:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=deleted
2024-05-08T01:17:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:17:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:17:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:17:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:17:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:17:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:17:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=10a9c7b6d76e
2024-05-08T01:17:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:17:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:17:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test deleted 10a9c7b6d76e
2024-05-08T01:17:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:17:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=10a9c7b6d76e
2024-05-08T01:17:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:17:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:17:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:17:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:17:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:17:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test: (2) No such file or directory
2024-05-08T01:17:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:17:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:17:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = deleted
2024-05-08T01:17:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ trash_restore cluster2 mirror test 10aa2fb4425e journal
2024-05-08T01:17:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=10aa2fb4425e
2024-05-08T01:17:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:17:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 trash restore mirror/10aa2fb4425e
2024-05-08T01:17:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ is_pool_mirror_mode_image mirror
2024-05-08T01:17:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' pool = image ']'
2024-05-08T01:17:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ case "${pool}" in
2024-05-08T01:17:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:17:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:17:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:17:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:17:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:17:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:17:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:17:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:17:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:17:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:17:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:17:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:17:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:17:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:17:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:17:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:17:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:17:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:17:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test
2024-05-08T01:17:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:17:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:17:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:17:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:17:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:17:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:17:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:17:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:17:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:17:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:17:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:17:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:17:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:17:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:17:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:17:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:17:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test
2024-05-08T01:17:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:17:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test
2024-05-08T01:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:17:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:17:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:17:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:17:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:17:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:17:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:17:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:17:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:17:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:17:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test
2024-05-08T01:17:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:17:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test
2024-05-08T01:17:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:17:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:17:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:17:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:17:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:17:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:17:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:17:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:17:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:17:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:17:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:17:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:17:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:17:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:17:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:17:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:17:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test
2024-05-08T01:17:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:17:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 1
2024-05-08T01:17:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:17:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:17:23.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:17:23.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:23.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test
2024-05-08T01:17:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:17:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:17:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:17:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:17:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:17:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:17:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:17:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:17:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:17:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:17:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:17:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:17:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:17:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:17:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:17:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:17:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test
2024-05-08T01:17:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:17:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test
2024-05-08T01:17:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:17:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:17:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:17:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:17:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:17:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:17:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:17:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:17:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:17:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:17:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:17:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:17:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:17:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:17:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:17:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:17:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test
2024-05-08T01:17:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:17:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test
2024-05-08T01:17:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:17:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:17:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:17:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:17:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:17:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:17:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:17:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:17:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:17:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:17:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:17:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:17:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:17:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:17:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:17:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:17:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test
2024-05-08T01:17:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:17:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 2
2024-05-08T01:17:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:17:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:17:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:17:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:17:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:17:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:17:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:17:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:17:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:17:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed (with rbd-mirror on one cluster)'
2024-05-08T01:17:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:17:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:17:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:17:25 TEST: check if removed 'images'\''' OMAP are removed '(with' rbd-mirror on one 'cluster)'
2024-05-08T01:17:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:17:25 TEST: check if removed images' OMAP are removed (with rbd-mirror on one cluster)
2024-05-08T01:17:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster2 mirror test
2024-05-08T01:17:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:17:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:17:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster2 mirror test
2024-05-08T01:17:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 snap purge mirror/test
2024-05-08T01:17:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rm mirror/test
2024-05-08T01:17:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:17:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_in_omap cluster1 mirror
2024-05-08T01:17:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global
2024-05-08T01:17:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local obj_name=rbd_mirroring
2024-05-08T01:17:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local filter=status_global
2024-05-08T01:17:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32
2024-05-08T01:17:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:17:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:17:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global
2024-05-08T01:17:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirroring
2024-05-08T01:17:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local filter=status_global
2024-05-08T01:17:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring
2024-05-08T01:17:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirroring
2024-05-08T01:17:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring
2024-05-08T01:17:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -c status_global
2024-05-08T01:17:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 0 = 0
2024-05-08T01:17:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=0
2024-05-08T01:17:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:17:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -eq 0 ']'
2024-05-08T01:17:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_
2024-05-08T01:17:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local obj_name=rbd_mirroring
2024-05-08T01:17:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local filter=image_
2024-05-08T01:17:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32
2024-05-08T01:17:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:17:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:17:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_
2024-05-08T01:17:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirroring
2024-05-08T01:17:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local filter=image_
2024-05-08T01:17:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring
2024-05-08T01:17:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirroring
2024-05-08T01:17:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -c image_
2024-05-08T01:17:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring
2024-05-08T01:17:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 0 = 0
2024-05-08T01:17:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=0
2024-05-08T01:17:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:17:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -eq 0 ']'
2024-05-08T01:17:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map
2024-05-08T01:17:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local obj_name=rbd_mirror_leader
2024-05-08T01:17:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local filter=image_map
2024-05-08T01:17:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32
2024-05-08T01:17:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:17:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:17:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map
2024-05-08T01:17:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirror_leader
2024-05-08T01:17:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local filter=image_map
2024-05-08T01:17:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader
2024-05-08T01:17:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -c image_map
2024-05-08T01:17:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirror_leader
2024-05-08T01:17:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader
2024-05-08T01:17:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 1 = 0
2024-05-08T01:17:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=1
2024-05-08T01:17:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:17:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 -eq 0 ']'
2024-05-08T01:17:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32
2024-05-08T01:17:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:17:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map
2024-05-08T01:17:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirror_leader
2024-05-08T01:17:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local filter=image_map
2024-05-08T01:17:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader
2024-05-08T01:17:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -c image_map
2024-05-08T01:17:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirror_leader
2024-05-08T01:17:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader
2024-05-08T01:17:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 1 = 0
2024-05-08T01:17:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=1
2024-05-08T01:17:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:17:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 -eq 0 ']'
2024-05-08T01:17:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32
2024-05-08T01:17:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:17:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:17:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map
2024-05-08T01:17:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirror_leader
2024-05-08T01:17:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local filter=image_map
2024-05-08T01:17:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader
2024-05-08T01:17:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -c image_map
2024-05-08T01:17:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirror_leader
2024-05-08T01:17:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader
2024-05-08T01:17:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 0 = 0
2024-05-08T01:17:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=0
2024-05-08T01:17:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:17:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -eq 0 ']'
2024-05-08T01:17:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_in_omap cluster2 mirror
2024-05-08T01:17:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global
2024-05-08T01:17:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local obj_name=rbd_mirroring
2024-05-08T01:17:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local filter=status_global
2024-05-08T01:17:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32
2024-05-08T01:17:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:17:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:17:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global
2024-05-08T01:17:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirroring
2024-05-08T01:17:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local filter=status_global
2024-05-08T01:17:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring
2024-05-08T01:17:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirroring
2024-05-08T01:17:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring
2024-05-08T01:17:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -c status_global
2024-05-08T01:17:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 0 = 0
2024-05-08T01:17:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=0
2024-05-08T01:17:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:17:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -eq 0 ']'
2024-05-08T01:17:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_
2024-05-08T01:17:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local obj_name=rbd_mirroring
2024-05-08T01:17:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local filter=image_
2024-05-08T01:17:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32
2024-05-08T01:17:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:17:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:17:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_
2024-05-08T01:17:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirroring
2024-05-08T01:17:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local filter=image_
2024-05-08T01:17:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring
2024-05-08T01:17:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirroring
2024-05-08T01:17:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring
2024-05-08T01:17:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -c image_
2024-05-08T01:17:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 0 = 0
2024-05-08T01:17:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=0
2024-05-08T01:17:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:17:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -eq 0 ']'
2024-05-08T01:17:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map
2024-05-08T01:17:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local obj_name=rbd_mirror_leader
2024-05-08T01:17:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local filter=image_map
2024-05-08T01:17:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32
2024-05-08T01:17:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:17:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:17:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map
2024-05-08T01:17:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirror_leader
2024-05-08T01:17:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local filter=image_map
2024-05-08T01:17:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader
2024-05-08T01:17:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirror_leader
2024-05-08T01:17:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader
2024-05-08T01:17:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -c image_map
2024-05-08T01:17:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:error getting omap key set mirror/rbd_mirror_leader: (2) No such file or directory
2024-05-08T01:17:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 0 = 0
2024-05-08T01:17:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=0
2024-05-08T01:17:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:17:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -eq 0 ']'
2024-05-08T01:17:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image_and_enable_mirror cluster2 mirror test journal
2024-05-08T01:17:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:17:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n journal ']'
2024-05-08T01:17:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image cluster2 mirror test
2024-05-08T01:17:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=128
2024-05-08T01:17:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:17:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test
2024-05-08T01:17:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ is_pool_mirror_mode_image mirror
2024-05-08T01:17:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' pool = image ']'
2024-05-08T01:17:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ case "${pool}" in
2024-05-08T01:17:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:17:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:17:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:17:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:17:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:17:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:17:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:17:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:17:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:17:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:17:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:17:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:17:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:17:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:17:30.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:30.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:17:30.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:17:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:17:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:17:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test
2024-05-08T01:17:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:17:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:17:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:17:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:17:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:17:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:17:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:17:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:17:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:17:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:17:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:17:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:17:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:17:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:17:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:17:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:17:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test
2024-05-08T01:17:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:17:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test
2024-05-08T01:17:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:17:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:17:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:17:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:17:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:17:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:17:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:17:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:17:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:17:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:17:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:17:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:17:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:17:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:17:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:17:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:17:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:30.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:30.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:30.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test
2024-05-08T01:17:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:17:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test
2024-05-08T01:17:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:17:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:17:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:17:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:17:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:17:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:17:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:17:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:17:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:17:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:17:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:17:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:17:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:17:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:17:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:17:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:17:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test
2024-05-08T01:17:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:17:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 1
2024-05-08T01:17:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:17:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:32.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:32.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:32.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:32.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:17:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:17:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test
2024-05-08T01:17:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:17:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:17:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:17:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:17:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:17:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:17:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:17:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:17:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:17:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:17:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:17:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:17:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:17:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:17:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:17:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:17:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:17:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test
2024-05-08T01:17:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:17:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test
2024-05-08T01:17:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:17:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:17:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:17:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:17:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:17:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:17:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:17:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:17:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:17:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:17:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:17:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:17:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:17:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:17:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:17:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:17:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:17:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test
2024-05-08T01:17:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:17:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test
2024-05-08T01:17:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:17:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:17:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:17:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:17:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:17:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:17:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:17:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:17:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:17:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:17:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:17:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:17:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:17:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:17:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:17:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:17:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:17:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test
2024-05-08T01:17:32.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:17:32.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:17:32.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:17:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 2
2024-05-08T01:17:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:17:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:17:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:17:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:17:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:17:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:17:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:17:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:17:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:17:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test 100
2024-05-08T01:17:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:17:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:17:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:17:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:17:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:17:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:17:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:17:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2500.03   bytes/sec: 9.8 MiB/s
2024-05-08T01:17:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:17:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:17:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:17:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:17:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:17:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:17:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:17:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:17:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:17:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:17:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:17:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:17:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:17:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:17:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:17:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:17:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:17:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:17:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:17:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:17:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:17:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:17:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:17:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:17:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:17:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:17:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:17:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:17:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:17:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:17:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:17:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:17:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:17:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:17:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:17:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:17:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:17:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:17:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:17:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:17:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:17:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:17:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:17:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:17:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:17:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:17:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:17:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:17:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:17:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:17:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:17:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:17:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:17:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:17:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:17:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:17:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:17:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=1'
2024-05-08T01:17:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=1, entry_tid=1'
2024-05-08T01:17:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=1' '!=' ''
2024-05-08T01:17:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:17:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=1' = ''
2024-05-08T01:17:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=1, tag_tid=1, entry_tid=1'
2024-05-08T01:17:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:17:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:17:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:17:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=2
2024-05-08T01:17:35.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=1'
2024-05-08T01:17:35.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:17:35.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:17:35.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=1
2024-05-08T01:17:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:17:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:17:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:17:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=101
2024-05-08T01:17:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=1'
2024-05-08T01:17:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:17:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:17:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=1
2024-05-08T01:17:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 2 = 1 -a 101 -le 1
2024-05-08T01:17:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:17:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:17:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:17:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:17:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:17:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:17:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:17:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:17:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:17:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:17:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:17:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:17:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:35.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:17:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:17:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:17:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:17:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:17:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:17:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:17:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:17:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:17:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:17:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:17:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:17:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:17:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:17:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:17:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:17:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:17:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:17:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:17:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:17:35.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:17:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:17:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:17:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:17:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:17:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:17:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:17:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:17:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:17:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:17:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:17:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:17:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:17:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:17:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:17:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:17:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:17:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:17:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:17:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:17:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:17:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:17:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:17:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:17:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:17:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:17:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:17:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:17:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}
2024-05-08T01:17:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:17:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:17:34'
2024-05-08T01:17:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:17:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:17:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:17:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}
2024-05-08T01:17:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:17:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:17:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:17:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:17:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:17:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:17:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:17:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:17:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:34'
2024-05-08T01:17:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:17:34'
2024-05-08T01:17:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:17:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:17:34'
2024-05-08T01:17:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:17:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:1}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:17:34</last_update></image></images></status>'
2024-05-08T01:17:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:17:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:17:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:34'
2024-05-08T01:17:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:17:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:17:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:17:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:17:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:17:36.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:36.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:36.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: failover and failback'
2024-05-08T01:17:36.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:17:36.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:17:36.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:17:36 TEST: failover and failback
2024-05-08T01:17:36.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:17:36 TEST: failover and failback
2024-05-08T01:17:36.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirrors cluster2
2024-05-08T01:17:36.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:36.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:17:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster2:0
2024-05-08T01:17:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:17:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:17:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:17:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:17:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:17:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:17:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:17:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:17:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:17:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:17:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:17:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster2:1
2024-05-08T01:17:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:17:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:17:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:17:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:17:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:17:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:17:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:17:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:17:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:17:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:17:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:17:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster2 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:17:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster2:2
2024-05-08T01:17:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:17:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:17:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:17:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:17:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:17:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:17:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:17:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:17:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:17:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:17:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:17:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster2 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:17:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster2:3
2024-05-08T01:17:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:17:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:17:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:17:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:17:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:17:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:17:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:17:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:17:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:17:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:17:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:17:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster2 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:17:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:17:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:17:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:17:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:17:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:17:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:17:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:17:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:17:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:17:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:17:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:17:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:17:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:17:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:17:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:17:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:17:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:17:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:17:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:17:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:17:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:17:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:17:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:17:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:17:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:17:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:17:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:17:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:17:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:17:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:17:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:17:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:17:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:17:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:17:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:17:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:17:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:17:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:17:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:40.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:40.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:40.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:40.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:17:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:17:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:17:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:17:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:17:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:17:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:17:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:17:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:17:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:17:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:17:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:17:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:17:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:17:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:17:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:17:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:17:39
2024-05-08T01:17:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:17:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:17:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:17:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:17:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:17:39'
2024-05-08T01:17:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:17:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:17:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:17:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:17:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:17:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:17:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:17:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:17:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:17:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:17:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:17:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:17:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:39'
2024-05-08T01:17:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:17:39'
2024-05-08T01:17:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:17:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:17:39'
2024-05-08T01:17:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:17:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:17:39</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:17:39</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:17:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:17:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:17:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:39'
2024-05-08T01:17:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:17:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:17:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:17:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:17:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:17:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:17:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:17:41.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:41.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:41.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:41.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:17:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:17:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:17:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:17:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:17:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:17:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:17:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:17:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:17:42
2024-05-08T01:17:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:17:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:17:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:17:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:17:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:17:39'
2024-05-08T01:17:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:17:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:17:42.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:17:42.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:17:42.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:17:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:17:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:17:42.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:17:42.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:17:42.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:17:42.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:17:42.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:17:42.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:42'
2024-05-08T01:17:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:17:42'
2024-05-08T01:17:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:17:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:17:42'
2024-05-08T01:17:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:17:42.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:17:42</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:17:39</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:17:42.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:17:42.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:17:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:42'
2024-05-08T01:17:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:17:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:17:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:17:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:17:42.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:17:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:17:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:17:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:17:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:17:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:17:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:17:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:17:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:17:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:17:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:17:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:17:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:17:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:17:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:17:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:17:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:17:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:17:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:17:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:17:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:17:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:17:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:17:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:17:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:17:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:17:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:17:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:17:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:17:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:17:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:17:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:17:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:17:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:17:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:17:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:17:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:17:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:17:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:17:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:17:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:17:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:17:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:17:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:17:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:17:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:17:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test 100
2024-05-08T01:17:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:17:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:17:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:17:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:17:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:17:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:17:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:17:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2272.75   bytes/sec: 8.9 MiB/s
2024-05-08T01:17:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:17:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:17:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:17:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:17:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:17:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:17:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:17:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:17:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:17:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:17:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:17:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:17:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:17:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:17:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:17:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:17:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:17:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:17:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:17:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:17:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:17:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:17:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:17:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:17:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:17:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:17:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:17:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:17:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:17:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:17:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:17:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:17:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:17:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:46.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:17:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:17:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:17:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected]
2024-05-08T01:17:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=5, entry_tid=0], [object_number=1, tag_tid=2, entry_tid=101], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:17:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:46.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=6, entry_tid=101'
2024-05-08T01:17:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:17:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:17:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:17:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:17:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:17:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:17:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:17:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:17:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:17:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:17:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected]
2024-05-08T01:17:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=5, entry_tid=0], [object_number=1, tag_tid=2, entry_tid=101], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:17:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=5, entry_tid=0'
2024-05-08T01:17:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=6, entry_tid=101' -a 'object_number=1, tag_tid=6, entry_tid=101' = 'object_number=0, tag_tid=5, entry_tid=0'
2024-05-08T01:17:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=5, entry_tid=0' '!=' ''
2024-05-08T01:17:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:17:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=5, entry_tid=0' = ''
2024-05-08T01:17:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=0, tag_tid=5, entry_tid=0'
2024-05-08T01:17:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=6, entry_tid=101'
2024-05-08T01:17:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:17:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:17:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=6
2024-05-08T01:17:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=5, entry_tid=0'
2024-05-08T01:17:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:17:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:17:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=5
2024-05-08T01:17:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=6, entry_tid=101'
2024-05-08T01:17:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:17:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:17:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=101
2024-05-08T01:17:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=5, entry_tid=0'
2024-05-08T01:17:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:17:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:17:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=0
2024-05-08T01:17:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 6 = 5 -a 101 -le 0
2024-05-08T01:17:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:17:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:17:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:17:47.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:17:47.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:17:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:17:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:17:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:17:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:17:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:17:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:17:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:17:47.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:47.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:17:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:17:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:17:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:17:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:17:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:17:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:17:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:17:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:17:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:17:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:17:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:17:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:17:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:17:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:17:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:17:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:17:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:17:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:17:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:17:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:17:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:17:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:17:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected]
2024-05-08T01:17:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=5, entry_tid=0], [object_number=1, tag_tid=2, entry_tid=101], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:17:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=6, entry_tid=101'
2024-05-08T01:17:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:17:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:17:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:17:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:17:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:17:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:17:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:17:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:17:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:17:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:17:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected]
2024-05-08T01:17:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=5, entry_tid=0], [object_number=1, tag_tid=2, entry_tid=101], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:17:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=5, entry_tid=0'
2024-05-08T01:17:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=6, entry_tid=101' -a 'object_number=1, tag_tid=6, entry_tid=101' = 'object_number=0, tag_tid=5, entry_tid=0'
2024-05-08T01:17:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=5, entry_tid=0' '!=' 'object_number=0, tag_tid=5, entry_tid=0'
2024-05-08T01:17:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:17:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.4
2024-05-08T01:17:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:17:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:17:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:17:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:17:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:17:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:17:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:17:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:17:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:17:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:17:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:17:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:17:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:17:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:17:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:17:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:17:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:17:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:17:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:17:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:17:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:17:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:17:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:17:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:17:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:17:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:17:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:17:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:17:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:17:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:17:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:17:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:17:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:17:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:17:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:17:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:17:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected]
2024-05-08T01:17:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected]
2024-05-08T01:17:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:48.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=6, entry_tid=101'
2024-05-08T01:17:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:17:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:17:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:17:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:17:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:17:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:17:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:17:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:17:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:17:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:17:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:17:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:17:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected]
2024-05-08T01:17:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected]
2024-05-08T01:17:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:17:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=1, tag_tid=6, entry_tid=101'
2024-05-08T01:17:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=6, entry_tid=101' -a 'object_number=1, tag_tid=6, entry_tid=101' = 'object_number=1, tag_tid=6, entry_tid=101'
2024-05-08T01:17:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:17:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:17:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:17:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:17:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:17:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:17:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:17:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:17:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:17:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:17:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:17:47
2024-05-08T01:17:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:17:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:17:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:17:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":5}}
2024-05-08T01:17:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:17:44'
2024-05-08T01:17:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:17:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:17:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:17:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:17:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:17:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:17:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:17:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:17:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:17:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:17:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:17:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:17:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:47'
2024-05-08T01:17:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:17:47'
2024-05-08T01:17:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:17:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:17:47'
2024-05-08T01:17:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:17:49.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:17:47</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:4},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:5}}</description><last_update>2024-05-08 01:17:44</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:17:49.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:17:49.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:17:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:47'
2024-05-08T01:17:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:17:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:17:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:17:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:17:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:17:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test
2024-05-08T01:17:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:17:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position
2024-05-08T01:17:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:17:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:17:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position ''
2024-05-08T01:17:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:17:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:17:50.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:50.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:50.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:50.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:17:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:17:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:17:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:17:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":5}}
2024-05-08T01:17:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:17:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:17:44
2024-05-08T01:17:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:17:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:17:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:17:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:17:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:17:47'
2024-05-08T01:17:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:17:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:17:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:17:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":5}}
2024-05-08T01:17:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:17:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:17:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:17:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:17:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:17:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:17:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:17:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:17:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:44'
2024-05-08T01:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:17:44'
2024-05-08T01:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:17:44'
2024-05-08T01:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:17:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:4},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:5}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:17:44</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:17:47</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:17:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:17:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:17:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:44'
2024-05-08T01:17:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:17:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:17:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:17:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:17:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:17:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror test
2024-05-08T01:17:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:17:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror-test.export
2024-05-08T01:17:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror-test.export
2024-05-08T01:17:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:17:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.export
2024-05-08T01:17:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror-test.export
2024-05-08T01:17:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror-test.export
2024-05-08T01:17:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:17:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:17:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:17:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export
2024-05-08T01:17:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:17:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:17:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:17:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:17:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:17:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:17:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:17:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:17:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:17:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:17:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:17:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:17:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:17:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:17:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:17:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:17:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:17:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:17:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:17:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:17:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:17:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:17:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:17:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:17:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:17:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:17:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:17:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:17:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:17:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:17:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:17:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:17:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:17:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:17:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:17:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:17:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:17:52.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:17:52.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:17:52.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:17:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:17:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:17:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:17:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:17:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:17:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:17:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:17:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:17:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:17:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:17:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:17:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:17:52
2024-05-08T01:17:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:17:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:17:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:17:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:17:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:17:52'
2024-05-08T01:17:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:17:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:17:53.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:53.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:17:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:17:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:17:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:17:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:17:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:17:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:17:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:17:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:17:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:17:54
2024-05-08T01:17:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:17:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:17:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:17:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:17:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:17:52'
2024-05-08T01:17:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:17:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:17:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:17:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:17:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:17:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:17:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:17:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:17:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:17:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:17:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:17:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:17:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:54'
2024-05-08T01:17:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:17:54'
2024-05-08T01:17:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:17:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:17:54'
2024-05-08T01:17:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:17:55.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:17:54</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:17:52</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:17:55.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:17:55.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:17:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:54'
2024-05-08T01:17:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:17:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:17:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:17:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:17:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:17:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:17:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:17:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:17:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:17:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:17:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:17:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:17:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:17:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:17:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:17:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:17:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:17:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:17:52
2024-05-08T01:17:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:17:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:17:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:17:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:17:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:17:54'
2024-05-08T01:17:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:17:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:17:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:17:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:17:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:17:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:17:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:17:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:17:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:17:56.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:17:56.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:17:56.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:17:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:52'
2024-05-08T01:17:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:17:52'
2024-05-08T01:17:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:17:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:17:52'
2024-05-08T01:17:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:17:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:17:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:17:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:17:52</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:17:54</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:17:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:17:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:17:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:52'
2024-05-08T01:17:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:17:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:17:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:17:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:17:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:17:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:17:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:17:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:17:56.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:17:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:17:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:17:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:17:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:17:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:17:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:17:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:17:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:17:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:17:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:17:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:17:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:17:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:17:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:17:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:17:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:17:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:17:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:17:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:17:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:17:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:17:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:17:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:17:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:17:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:17:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:17:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:17:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:17:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:17:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:17:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:17:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:17:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:17:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:17:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:17:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:17:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:17:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:17:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:17:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:17:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:17:57.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:17:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:17:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:17:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:17:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:17:57.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:17:57.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:57.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:57.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:17:57.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:17:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:17:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:17:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:17:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:17:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:17:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:17:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:17:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:17:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:17:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:17:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:17:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:17:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:17:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:17:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:17:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:17:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:17:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:17:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:17:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:17:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:17:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:17:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:17:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:17:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:17:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:17:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:17:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:17:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:17:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:17:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:17:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:17:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:17:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:17:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:17:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:17:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:17:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:17:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:17:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:17:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:17:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:17:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:17:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:17:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:17:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:17:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:17:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:17:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:17:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:17:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:17:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:17:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:17:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:17:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:17:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:17:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:17:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:17:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:17:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:17:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:17:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:17:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:17:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:17:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:18:01.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:18:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:18:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:18:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:18:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:18:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:18:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:18:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:18:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:18:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:18:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:18:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:18:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:18:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:18:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:18:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:18:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:18:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:18:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:18:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:18:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:18:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:18:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:18:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:18:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:18:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:18:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:18:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:18:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:18:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:18:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:18:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:18:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:18:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:18:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:18:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:18:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:18:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:18:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:18:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:18:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:18:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:18:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:18:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:18:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:18:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:17:59
2024-05-08T01:18:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:18:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:18:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:18:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:02'
2024-05-08T01:18:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:18:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:18:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:18:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:18:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:18:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:18:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:18:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:18:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:18:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:18:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:59'
2024-05-08T01:18:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:17:59'
2024-05-08T01:18:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:17:59'
2024-05-08T01:18:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:18:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:17:59</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:18:02</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:18:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:18:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:18:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:17:59'
2024-05-08T01:18:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:18:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:18:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:18:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:18:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:18:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:18:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:18:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:18:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:02
2024-05-08T01:18:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:18:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:18:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:18:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:17:59'
2024-05-08T01:18:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:18:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:18:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:18:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:18:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:18:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:18:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:18:03.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:18:03.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:18:03.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:18:03.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:02'
2024-05-08T01:18:03.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:18:02'
2024-05-08T01:18:03.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:03.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:18:02'
2024-05-08T01:18:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:18:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:18:02</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:17:59</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:18:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:18:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:18:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:02'
2024-05-08T01:18:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:18:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:18:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:18:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:18:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:18:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:18:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:18:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:18:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:18:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:18:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:18:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:18:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:18:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:18:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:18:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:18:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:18:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:18:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:18:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:18:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:18:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:18:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:18:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:18:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:18:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:18:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:18:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:18:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:18:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:18:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:18:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:18:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:18:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:18:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:18:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:18:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:18:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:18:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:18:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:18:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:18:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:18:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:18:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:18:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:18:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:18:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:18:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:18:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:18:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:18:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:18:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:18:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:18:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:18:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:18:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:18:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:18:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:18:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:18:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:18:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:18:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:18:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:18:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:18:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:18:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:18:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:18:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:18:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:18:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:18:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:18:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:18:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:18:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:18:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:18:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:18:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:18:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:18:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:18:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:18:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:18:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:18:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:04.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:18:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:18:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:18:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:18:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:18:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:18:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:18:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:18:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:18:04.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:04.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:04.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:04.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:18:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:18:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:18:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:18:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:18:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:18:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:18:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:18:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:18:05.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:18:05.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected]
2024-05-08T01:18:05.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected]
2024-05-08T01:18:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:05.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=11, entry_tid=0'
2024-05-08T01:18:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:18:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:18:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:18:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:18:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:18:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:18:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:18:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:18:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:18:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected]
2024-05-08T01:18:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected]
2024-05-08T01:18:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=11, entry_tid=0'
2024-05-08T01:18:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=11, entry_tid=0' -a 'object_number=0, tag_tid=11, entry_tid=0' = 'object_number=0, tag_tid=11, entry_tid=0'
2024-05-08T01:18:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test
2024-05-08T01:18:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:18:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position
2024-05-08T01:18:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:18:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:18:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:05.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:05.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position ''
2024-05-08T01:18:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:18:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:18:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:18:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:18:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":8},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}}
2024-05-08T01:18:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:04
2024-05-08T01:18:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:18:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:18:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:18:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:02'
2024-05-08T01:18:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:18:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:18:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:18:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":8},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}}
2024-05-08T01:18:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:18:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:18:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:18:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:18:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:18:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:18:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:18:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:18:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:04'
2024-05-08T01:18:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:18:04'
2024-05-08T01:18:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:18:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:18:04'
2024-05-08T01:18:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:18:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:8},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:11}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:18:04</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:18:02</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:18:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:18:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:18:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:04'
2024-05-08T01:18:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:18:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:18:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:18:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:18:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:18:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:18:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:18:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:18:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:18:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:18:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:18:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:18:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:07
2024-05-08T01:18:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:18:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:18:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":8},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}}
2024-05-08T01:18:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:04'
2024-05-08T01:18:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:18:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:18:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:18:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:18:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:18:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:18:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:18:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:18:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:18:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:18:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:18:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:18:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:07'
2024-05-08T01:18:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:18:07'
2024-05-08T01:18:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:18:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:18:07'
2024-05-08T01:18:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:18:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:18:07</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:8},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:11}}</description><last_update>2024-05-08 01:18:04</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:18:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:18:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:18:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:07'
2024-05-08T01:18:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:18:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:18:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:18:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:18:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:18:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror test
2024-05-08T01:18:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:18:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror-test.export
2024-05-08T01:18:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror-test.export
2024-05-08T01:18:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.export
2024-05-08T01:18:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror-test.export
2024-05-08T01:18:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror-test.export
2024-05-08T01:18:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:18:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:18:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export
2024-05-08T01:18:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:18:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:18:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:18:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:18:07.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:18:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:18:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:18:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:18:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:18:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:18:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:18:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:18:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:08.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:18:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:18:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:18:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:18:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:18:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:18:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:18:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:18:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:18:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:18:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:18:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:18:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:18:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:18:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:18:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:18:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:18:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:18:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:18:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:18:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:18:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:18:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:18:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:18:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:18:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:18:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:18:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:18:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:18:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:18:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:18:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:18:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:18:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:18:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:18:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:18:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:18:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:18:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:18:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:18:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:18:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:18:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:18:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:18:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:18:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:18:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:18:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:18:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:18:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:18:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:18:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:18:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:18:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:18:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:18:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:18:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:18:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:18:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:18:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:18:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:18:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:18:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:18:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:18:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:18:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:18:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:18:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:18:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:18:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:18:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:18:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:18:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:18:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:18:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:18:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:18:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:18:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:18:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:18:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:18:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:18:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:18:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:18:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:18:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:18:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:18:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:18:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:18:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:09
2024-05-08T01:18:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:18:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:18:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:18:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:12'
2024-05-08T01:18:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:18:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:18:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:18:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:18:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:18:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:18:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:18:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:14
2024-05-08T01:18:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:18:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:18:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:18:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:12'
2024-05-08T01:18:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:18:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:18:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:18:14.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:18:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:18:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:18:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:18:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:18:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:18:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:18:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:14'
2024-05-08T01:18:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:18:14'
2024-05-08T01:18:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:18:14'
2024-05-08T01:18:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:18:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:18:14</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:18:12</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:18:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:18:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:18:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:14'
2024-05-08T01:18:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:18:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:18:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:18:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:18:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:18:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:18:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:18:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:18:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:12
2024-05-08T01:18:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:18:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:18:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:18:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:14'
2024-05-08T01:18:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:18:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:18:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:18:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:18:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:18:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:18:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:18:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:18:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:18:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:18:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:12'
2024-05-08T01:18:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:18:12'
2024-05-08T01:18:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:18:12'
2024-05-08T01:18:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:18:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:18:12</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:18:14</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:18:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:18:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:18:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:12'
2024-05-08T01:18:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:18:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:18:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:18:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:18:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:18:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:18:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:18:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:18:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:18:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:18:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:18:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:18:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:18:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:18:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:18:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:18:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:18:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:18:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:18:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:18:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:18:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:18:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:18:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:18:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:18:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:18:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:18:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:18:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:18:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:18:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:18:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:18:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:18:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:18:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:18:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:18:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:18:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:18:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:18:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:18:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:18:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:18:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:18:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:18:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:18:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:18:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:18:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:18:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:18:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:18:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:18:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:18:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:18:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:18:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:18:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:18:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:18:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:18:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:18:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:18:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:18:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:18:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:18:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:18:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:18:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:18:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:18:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:18:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:18:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:18:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:18:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:18:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:18:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:18:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:18:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:18:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:18:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:18:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:18:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:18:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:18:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:18:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:18:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:18:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:18:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:18:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:18.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:18.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:18.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:18:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:18:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:18:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:18:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:18:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:18:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:18:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:18:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster1 mirror test 100
2024-05-08T01:18:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:18:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:18:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:18:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:18:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:18:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:18:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:18:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 1562.52   bytes/sec: 6.1 MiB/s
2024-05-08T01:18:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:18:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:18:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:18:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:18:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:18:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:18:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:18:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:18:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:18:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:18:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:18:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:18:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:18:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:18:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:18:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:18:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:18:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:18:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:18:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:18:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:18:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:18:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:18:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:18:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:18:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:18:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:18:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:18:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:18:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:18:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:18:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:18:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:18:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:18:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:18:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:18:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:18:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:18:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:18:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:18:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:18:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:18:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:18:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:18:19.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:19.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:19.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:19.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:18:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:18:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:18:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:18:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:18:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:18:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:18:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:18:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:18:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:18:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:18:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=12, entry_tid=0]]], state=connected]
2024-05-08T01:18:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=13, entry_tid=101'
2024-05-08T01:18:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:18:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:18:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:18:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:18:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:18:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:18:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:18:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:18:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:18:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:18:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=12, entry_tid=0]]], state=connected]
2024-05-08T01:18:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos=
2024-05-08T01:18:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=13, entry_tid=101' -a 'object_number=1, tag_tid=13, entry_tid=101' = ''
2024-05-08T01:18:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test '' '!=' ''
2024-05-08T01:18:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:18:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.4
2024-05-08T01:18:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:18:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:18:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:18:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:18:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:18:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:18:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:18:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:18:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:18:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:18:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:18:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:18:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:18:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:18:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:18:20.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:18:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:18:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:18:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:18:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:18:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:18:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:18:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:18:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:18:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:18:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:18:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:18:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:18:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:18:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:18:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:18:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:18:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:18:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:18:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=12, entry_tid=0]]], state=connected]
2024-05-08T01:18:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=13, entry_tid=101'
2024-05-08T01:18:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:18:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:18:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:18:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:18:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:18:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:18:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:18:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:18:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:18:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:18:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=12, entry_tid=0]]], state=connected]
2024-05-08T01:18:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos=
2024-05-08T01:18:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=13, entry_tid=101' -a 'object_number=1, tag_tid=13, entry_tid=101' = ''
2024-05-08T01:18:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test '' '!=' ''
2024-05-08T01:18:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:18:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.8
2024-05-08T01:18:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:18:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:18:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:18:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:18:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:18:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:18:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:18:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:18:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:18:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:18:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:18:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:18:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:18:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:18:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:18:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:18:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:18:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:18:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:18:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:18:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:18:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:18:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:18:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:18:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:18:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:18:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:18:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:18:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:18:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:18:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:18:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:18:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:18:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:18:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:18:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:18:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:18:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:18:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:18:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:18:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:18:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:18:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:18:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:18:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:18:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:18:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=13, entry_tid=101'
2024-05-08T01:18:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:18:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:18:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:18:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:18:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:18:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:18:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:18:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:18:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:18:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:18:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:18:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=1, tag_tid=13, entry_tid=101'
2024-05-08T01:18:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=13, entry_tid=101' -a 'object_number=1, tag_tid=13, entry_tid=101' = 'object_number=1, tag_tid=13, entry_tid=101'
2024-05-08T01:18:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:18:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:18:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:18:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:18:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:18:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:18:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:18:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:19
2024-05-08T01:18:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:18:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:18:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":9},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":12}}
2024-05-08T01:18:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:17'
2024-05-08T01:18:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:18:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:18:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:18:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:18:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:18:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:18:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:18:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:18:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:18:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:18:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:18:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:18:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:19'
2024-05-08T01:18:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:18:19'
2024-05-08T01:18:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:18:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:18:19'
2024-05-08T01:18:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:18:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:18:19</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:9},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:12}}</description><last_update>2024-05-08 01:18:17</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:18:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:18:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:18:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:19'
2024-05-08T01:18:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:18:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:18:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:18:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:18:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:18:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster2 mirror test
2024-05-08T01:18:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:18:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying primary_position
2024-05-08T01:18:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:18:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:18:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying primary_position ''
2024-05-08T01:18:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:18:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:18:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:18:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:18:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":9},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":12}}
2024-05-08T01:18:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:17
2024-05-08T01:18:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:18:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:18:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:18:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:19'
2024-05-08T01:18:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:18:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:18:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:18:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":9},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":12}}
2024-05-08T01:18:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:18:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:18:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:18:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:18:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:18:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:18:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:18:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:18:23.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:17'
2024-05-08T01:18:23.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:18:17'
2024-05-08T01:18:23.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:23.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:23.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:23.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:18:23.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:18:17'
2024-05-08T01:18:23.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:23.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:23.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:18:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:9},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:12}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:18:17</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:18:19</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:18:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:18:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:18:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:17'
2024-05-08T01:18:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:18:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:18:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:18:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:18:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror test
2024-05-08T01:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror-test.export
2024-05-08T01:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror-test.export
2024-05-08T01:18:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.export
2024-05-08T01:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror-test.export
2024-05-08T01:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror-test.export
2024-05-08T01:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:18:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:18:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export
2024-05-08T01:18:24.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:18:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:18:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:18:24.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:18:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:18:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:18:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:18:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:18:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:18:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:18:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:18:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:18:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:18:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:18:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:18:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:18:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:18:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:18:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:18:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:18:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:18:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:18:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:18:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:18:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:18:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:18:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:18:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:18:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:18:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:18:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:18:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:18:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:18:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:18:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:18:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:18:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:18:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:18:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:18:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:18:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:18:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:18:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:18:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:18:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:18:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:18:25.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:25.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:18:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:18:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:18:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:18:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:18:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:18:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:18:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:18:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:18:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:18:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:18:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:18:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:18:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:18:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:18:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:18:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:18:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:18:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:18:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:18:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:18:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:18:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:18:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:18:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:18:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:18:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:18:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:18:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:18:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:18:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:18:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:18:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:18:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:18:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:18:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:18:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:18:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:18:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:18:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:18:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:18:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:18:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:18:27.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:18:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:18:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:18:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:18:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:18:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:18:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:18:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:18:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:18:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:18:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:18:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:24
2024-05-08T01:18:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:18:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:18:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:18:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:27'
2024-05-08T01:18:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:18:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:18:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:18:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:18:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:30.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:30.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:18:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:18:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:18:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:29
2024-05-08T01:18:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:18:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:18:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:18:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:27'
2024-05-08T01:18:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:18:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:18:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:18:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:18:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:18:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:18:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:18:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:18:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:18:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:18:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:29'
2024-05-08T01:18:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:18:29'
2024-05-08T01:18:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:18:29'
2024-05-08T01:18:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:18:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:18:29</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:18:27</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:18:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:18:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:18:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:29'
2024-05-08T01:18:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:18:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:18:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:18:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:18:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:18:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:18:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:18:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:18:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:27
2024-05-08T01:18:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:18:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:18:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:18:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:29'
2024-05-08T01:18:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:18:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:18:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:18:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:18:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:18:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:18:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:18:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:18:32.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:18:32.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:18:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:27'
2024-05-08T01:18:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:18:27'
2024-05-08T01:18:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:18:27'
2024-05-08T01:18:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:18:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:18:32</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:18:29</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:18:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:18:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:18:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:32'
2024-05-08T01:18:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:18:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:18:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:18:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:18:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:18:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:18:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:18:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:18:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:18:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:18:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:18:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:18:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:18:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:18:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:18:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:18:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:18:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:18:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:18:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:18:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:18:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:18:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:18:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:18:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:18:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:18:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:18:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:18:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:18:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:18:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:18:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:18:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:18:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:18:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:18:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:18:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:18:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:18:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:18:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:18:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:18:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:18:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:18:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:18:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:18:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:18:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:18:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:18:33.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:18:33.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:18:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:18:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:18:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:18:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:18:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:18:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:18:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:18:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:18:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:18:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:18:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:18:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:18:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:18:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:18:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:18:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:18:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:18:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:18:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:18:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:18:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:18:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:18:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:18:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:18:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:18:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:18:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:18:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:18:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:18:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:18:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:18:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:18:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:18:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:18:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:18:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:18:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:18:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:18:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:18:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:18:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:18:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test 100
2024-05-08T01:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:18:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:18:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:18:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:18:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:18:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:18:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:18:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2500.02   bytes/sec: 9.8 MiB/s
2024-05-08T01:18:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:18:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:18:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:18:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:18:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:18:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:18:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:18:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:18:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:18:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:18:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:18:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:18:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:18:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:18:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:18:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:18:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:18:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:18:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:18:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:18:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:18:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:18:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:18:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:18:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:18:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:18:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:18:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:18:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:18:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:18:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:18:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:18:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:18:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:18:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:18:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:18:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:18:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:35.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:35.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:18:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:18:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:18:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:18:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:18:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:18:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:18:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:18:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:18:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:18:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:18:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected]
2024-05-08T01:18:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=18, entry_tid=101'
2024-05-08T01:18:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:18:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:18:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:18:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:18:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:18:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:18:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:18:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:18:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:18:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:18:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected]
2024-05-08T01:18:36.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=17, entry_tid=0'
2024-05-08T01:18:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=18, entry_tid=101' -a 'object_number=1, tag_tid=18, entry_tid=101' = 'object_number=0, tag_tid=17, entry_tid=0'
2024-05-08T01:18:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=17, entry_tid=0' '!=' ''
2024-05-08T01:18:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:18:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=17, entry_tid=0' = ''
2024-05-08T01:18:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=0, tag_tid=17, entry_tid=0'
2024-05-08T01:18:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=18, entry_tid=101'
2024-05-08T01:18:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:18:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:18:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=18
2024-05-08T01:18:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=17, entry_tid=0'
2024-05-08T01:18:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:18:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:18:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=17
2024-05-08T01:18:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=18, entry_tid=101'
2024-05-08T01:18:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:18:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:18:36.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=101
2024-05-08T01:18:36.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=17, entry_tid=0'
2024-05-08T01:18:36.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:18:36.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:18:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=0
2024-05-08T01:18:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 18 = 17 -a 101 -le 0
2024-05-08T01:18:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:18:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:18:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:18:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:18:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:18:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:18:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:18:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:18:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:18:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:18:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:18:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:18:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:18:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:18:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:18:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:18:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:18:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:18:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:18:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:18:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:18:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:18:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:18:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:18:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:18:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:18:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:18:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:18:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:18:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:18:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:18:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:18:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:18:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:18:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:18:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:18:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:18:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:18:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:18:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:18:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:18:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:18:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:18:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:18:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:18:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:18:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:18:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:18:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected]
2024-05-08T01:18:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=18, entry_tid=101'
2024-05-08T01:18:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:18:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:18:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:18:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:18:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:18:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:18:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:18:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:18:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:18:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:18:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected]
2024-05-08T01:18:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=17, entry_tid=0'
2024-05-08T01:18:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=18, entry_tid=101' -a 'object_number=1, tag_tid=18, entry_tid=101' = 'object_number=0, tag_tid=17, entry_tid=0'
2024-05-08T01:18:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=17, entry_tid=0' '!=' 'object_number=0, tag_tid=17, entry_tid=0'
2024-05-08T01:18:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:18:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.4
2024-05-08T01:18:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:18:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:18:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:18:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:18:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:18:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:18:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:18:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:18:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:18:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:18:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:18:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:18:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:18:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:18:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:18:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:18:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:18:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:18:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:18:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:18:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:18:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:18:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:18:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:18:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:18:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:18:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:18:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:18:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:18:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:18:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:18:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:18:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:18:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:18:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:18:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:18:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:18:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:18:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:18:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:18:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:18:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:18:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:18:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:18:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:18:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:18:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=18, entry_tid=101'
2024-05-08T01:18:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:18:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:18:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:18:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:18:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:18:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:18:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:18:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:18:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:18:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:18:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:18:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=1, tag_tid=18, entry_tid=101'
2024-05-08T01:18:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=18, entry_tid=101' -a 'object_number=1, tag_tid=18, entry_tid=101' = 'object_number=1, tag_tid=18, entry_tid=101'
2024-05-08T01:18:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test
2024-05-08T01:18:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:18:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position
2024-05-08T01:18:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:18:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:18:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position ''
2024-05-08T01:18:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:18:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:18:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:18:38.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:38.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:38.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:18:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":13},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}}
2024-05-08T01:18:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:34
2024-05-08T01:18:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:18:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:18:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:18:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:37'
2024-05-08T01:18:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:18:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:18:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:18:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":13},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}}
2024-05-08T01:18:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:18:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:18:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:18:38.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:18:38.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:18:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:18:38.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:18:38.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:18:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:34'
2024-05-08T01:18:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:18:34'
2024-05-08T01:18:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:18:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:18:34'
2024-05-08T01:18:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:18:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:13},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:17}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:18:34</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:18:37</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:18:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:18:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:18:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:34'
2024-05-08T01:18:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:18:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:18:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:18:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:18:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:18:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:18:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:18:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:18:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:18:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:18:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:18:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:18:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:37
2024-05-08T01:18:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:18:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:18:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":13},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}}
2024-05-08T01:18:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:34'
2024-05-08T01:18:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:18:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:18:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:18:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:18:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:18:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:18:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:18:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:18:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:18:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:18:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:18:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:18:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:37'
2024-05-08T01:18:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:18:37'
2024-05-08T01:18:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:18:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:18:37'
2024-05-08T01:18:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:18:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:18:37</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:13},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:17}}</description><last_update>2024-05-08 01:18:34</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:18:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:18:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:18:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:37'
2024-05-08T01:18:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:18:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:18:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:18:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:18:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:18:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror test
2024-05-08T01:18:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:18:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror-test.export
2024-05-08T01:18:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror-test.export
2024-05-08T01:18:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.export
2024-05-08T01:18:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror-test.export
2024-05-08T01:18:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror-test.export
2024-05-08T01:18:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:18:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:18:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export
2024-05-08T01:18:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:18:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:18:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:18:39.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:18:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:18:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: failover / failback loop'
2024-05-08T01:18:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:18:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:18:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:18:40 TEST: failover / failback loop
2024-05-08T01:18:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:18:40 TEST: failover / failback loop
2024-05-08T01:18:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 1 20
2024-05-08T01:18:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:18:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:18:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:18:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:18:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:18:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:18:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:18:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:18:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:18:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:18:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:18:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:18:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:18:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:18:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:18:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:18:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:18:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:18:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:18:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:18:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:18:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:18:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:18:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:18:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:18:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:18:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:18:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:18:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:18:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:18:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:18:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:18:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:18:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:18:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:18:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:18:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:18:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:18:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:18:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:18:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:18:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:18:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:18:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:18:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:18:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:18:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:18:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:18:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:18:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:18:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:18:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:18:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:18:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:41
2024-05-08T01:18:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:18:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:18:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:18:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:42'
2024-05-08T01:18:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:18:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:18:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:18:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:18:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:18:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:18:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:18:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:44
2024-05-08T01:18:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:18:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:18:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:18:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:42'
2024-05-08T01:18:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:18:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:18:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:18:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:18:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:18:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:18:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:18:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:18:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:18:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:18:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:44'
2024-05-08T01:18:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:18:44'
2024-05-08T01:18:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:18:44'
2024-05-08T01:18:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:18:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:18:44</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:18:42</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:18:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:18:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:18:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:44'
2024-05-08T01:18:44.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:18:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:18:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:18:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:18:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:18:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:18:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:18:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:18:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:42
2024-05-08T01:18:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:18:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:18:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:18:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:44'
2024-05-08T01:18:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:18:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:18:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:18:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:18:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:18:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:18:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:18:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:18:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:18:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:18:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:42'
2024-05-08T01:18:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:18:42'
2024-05-08T01:18:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:18:42'
2024-05-08T01:18:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:18:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:18:42</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:18:44</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:18:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:18:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:18:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:42'
2024-05-08T01:18:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:18:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:18:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:18:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:18:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:18:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:18:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:18:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:18:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:18:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:18:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:18:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:18:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:18:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:18:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:18:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:18:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:18:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:18:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:18:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:18:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:18:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:18:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:18:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:18:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:18:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:18:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:18:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:18:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:18:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:18:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:18:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:18:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:18:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:18:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:18:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:18:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:18:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:18:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:18:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:18:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:18:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:18:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:18:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:18:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:18:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:18:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:18:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:18:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:18:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:18:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:18:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:18:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:18:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:18:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:18:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:18:49.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:18:49.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:18:49.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:18:49.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:49.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:18:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:18:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:18:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:18:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:18:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:18:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:18:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:18:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:18:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:18:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:18:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:18:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:18:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:18:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:18:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:18:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:18:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:18:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:18:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:18:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:18:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:18:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:18:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:18:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:18:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:18:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:18:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:18:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:18:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:18:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:18:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:18:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:18:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:18:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:18:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:18:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:18:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:18:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:18:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:18:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:18:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:18:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:18:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:18:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:18:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:18:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:18:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:18:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:18:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:18:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:18:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:18:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:18:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:18:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:18:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:18:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:18:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:18:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:18:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:18:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:18:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:18:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:18:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:18:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:18:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:18:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:18:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:18:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:18:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:18:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:18:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:18:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:18:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:18:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:18:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:18:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:18:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:18:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:18:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:18:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:18:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:18:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:18:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:18:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:18:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:18:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:18:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:18:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:18:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=19, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:18:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=19, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:18:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=19, entry_tid=0'
2024-05-08T01:18:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:18:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:18:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:18:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:18:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:18:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:18:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:18:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:18:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:18:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:18:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:18:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:18:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=19, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:18:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=19, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:18:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:18:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=19, entry_tid=0'
2024-05-08T01:18:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=19, entry_tid=0' -a 'object_number=0, tag_tid=19, entry_tid=0' = 'object_number=0, tag_tid=19, entry_tid=0'
2024-05-08T01:18:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:18:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:18:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:50.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:18:50.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:50.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:18:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:18:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:18:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:18:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:49
2024-05-08T01:18:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:18:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:18:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":15},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19}}
2024-05-08T01:18:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:47'
2024-05-08T01:18:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:18:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:18:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:18:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:18:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:18:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:18:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:18:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:18:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:18:50.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:18:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:18:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:18:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:49'
2024-05-08T01:18:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:18:49'
2024-05-08T01:18:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:18:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:18:49'
2024-05-08T01:18:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:18:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:18:49</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:15},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:19}}</description><last_update>2024-05-08 01:18:47</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:18:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:18:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:18:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:49'
2024-05-08T01:18:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:18:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:18:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:18:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:18:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:18:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:18:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:18:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:18:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:18:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:18:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:18:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":15},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19}}
2024-05-08T01:18:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:47
2024-05-08T01:18:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:51.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:18:51.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:18:51.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:18:51.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:49'
2024-05-08T01:18:51.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:18:51.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:18:51.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:18:51.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":15},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19}}
2024-05-08T01:18:51.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:18:51.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:18:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:18:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:18:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:18:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:18:51.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:18:51.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:18:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:47'
2024-05-08T01:18:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:18:47'
2024-05-08T01:18:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:18:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:18:47'
2024-05-08T01:18:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:18:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:15},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:19}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:18:47</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:18:49</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:18:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:18:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:18:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:47'
2024-05-08T01:18:52.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:18:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:18:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:18:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:18:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:18:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:18:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:18:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:18:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:18:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:18:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:18:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:18:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:18:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:18:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:18:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:18:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:18:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:18:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:18:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:18:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:18:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:18:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:18:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:18:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:18:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:18:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:18:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:18:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:18:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:18:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:18:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:18:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:18:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:18:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:18:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:18:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:18:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:18:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:18:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:18:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:18:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:18:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:18:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:18:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:18:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:18:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:18:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:18:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:18:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:18:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:18:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:18:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:18:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:18:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:18:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:18:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:18:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:18:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:18:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:18:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:18:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:18:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:18:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:18:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:18:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:18:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:18:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:18:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:18:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:18:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:18:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:18:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:18:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:18:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:18:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:18:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:18:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:18:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:18:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:18:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:18:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:18:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:18:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:18:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:18:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:18:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:18:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:18:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:18:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:18:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:18:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:18:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:18:55.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:18:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:18:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:18:55.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:18:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:18:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:18:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:18:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:18:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:18:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:18:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:54
2024-05-08T01:18:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:18:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:18:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:18:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:53'
2024-05-08T01:18:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:18:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:18:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:18:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:18:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:18:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:18:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:18:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:18:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:18:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:18:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:54'
2024-05-08T01:18:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:18:54'
2024-05-08T01:18:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:18:54'
2024-05-08T01:18:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:18:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:18:54</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>stopped</description><last_update>2024-05-08 01:18:53</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:18:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:18:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:18:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:54'
2024-05-08T01:18:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:18:56.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:18:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:18:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:18:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:18:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:18:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:18:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:18:57.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:18:57.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:18:57.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:18:57.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:18:57.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:18:57.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:57
2024-05-08T01:18:57.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:18:57.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:18:57.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:18:57.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:18:57.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:54'
2024-05-08T01:18:57.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:18:57.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:18:57.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:18:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:18:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:18:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:18:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:18:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:18:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:18:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:18:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:57'
2024-05-08T01:18:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:18:57'
2024-05-08T01:18:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:18:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:18:57'
2024-05-08T01:18:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:18:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:18:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:18:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:18:57</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:18:54</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:18:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:18:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:18:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:57'
2024-05-08T01:18:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:18:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:18:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:18:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:18:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:18:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:18:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:18:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:18:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:18:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:18:57.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:18:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:18:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:18:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:18:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:18:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:18:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:18:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:18:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:18:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:18:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:18:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:18:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:18:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:18:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:18:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:18:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:18:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:18:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:18:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:18:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:18:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:18:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:18:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:18:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:18:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:18:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:18:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:18:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:18:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:18:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:18:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:18:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:18:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:18:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:18:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:18:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:18:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:18:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:18:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:18:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:18:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:18:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:18:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:18:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:18:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:18:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:18:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:18:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:18:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:18:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:18:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:18:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:18:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:18:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:18:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:18:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:18:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:18:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:18:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:18:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:19:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:19:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:19:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:19:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:19:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:19:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:19:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:19:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:19:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:19:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:19:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:19:01.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:19:01.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:19:01.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:19:01.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:19:01.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:19:01.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:01.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:19:01.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:19:01.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:19:01.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:19:01.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:19:01.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:19:01.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:01.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:19:01.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:19:01.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:19:01.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:19:01.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:19:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:19:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:19:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:19:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:19:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:19:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:19:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:19:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:19:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:19:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:19:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:19:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:19:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:19:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:19:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:19:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:19:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:19:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:19:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:19:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:19:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:19:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:19:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:19:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:19:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:19:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:19:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:19:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:19:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:19:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:19:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:19:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:19:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:19:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:19:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:19:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:19:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:19:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:19:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:19:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:19:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:19:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:19:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:19:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:19:01.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:19:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:19:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:19:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:19:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:19:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:19:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:19:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:19:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:19:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:19:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:19:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:19:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:19:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:19:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:19:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:19:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:19:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:19:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:19:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:19:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:19:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:19:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:19:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:19:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:19:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:19:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:19:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:19:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:19:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:19:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:19:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:19:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:19:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:19:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:19:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:19:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:19:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:19:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:19:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:19:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:19:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:19:01.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:01.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=23, entry_tid=0'
2024-05-08T01:19:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:19:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:19:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:19:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:19:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:19:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:19:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:19:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:19:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:19:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:19:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:19:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:19:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=23, entry_tid=0'
2024-05-08T01:19:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=23, entry_tid=0' -a 'object_number=0, tag_tid=23, entry_tid=0' = 'object_number=0, tag_tid=23, entry_tid=0'
2024-05-08T01:19:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:19:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:19:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:19:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:19:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:19:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:19:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:19:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:02
2024-05-08T01:19:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:19:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:19:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":20},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}}
2024-05-08T01:19:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:18:59'
2024-05-08T01:19:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:19:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:19:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:19:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:19:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:19:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:19:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:19:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:19:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:19:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:19:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:19:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:19:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:02'
2024-05-08T01:19:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:19:02'
2024-05-08T01:19:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:19:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:19:02'
2024-05-08T01:19:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:19:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:19:02</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:20},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:23}}</description><last_update>2024-05-08 01:18:59</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:19:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:19:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:19:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:02'
2024-05-08T01:19:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:19:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:19:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:19:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:19:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:19:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:19:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:19:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:19:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:19:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:19:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:19:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":20},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}}
2024-05-08T01:19:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:18:59
2024-05-08T01:19:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:19:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:19:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:19:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:02'
2024-05-08T01:19:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:19:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:19:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:19:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":20},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}}
2024-05-08T01:19:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:19:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:19:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:19:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:19:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:19:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:19:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:19:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:19:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:59'
2024-05-08T01:19:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:18:59'
2024-05-08T01:19:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:19:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:18:59'
2024-05-08T01:19:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:19:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:20},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:23}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:18:59</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:19:02</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:19:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:19:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:19:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:18:59'
2024-05-08T01:19:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:19:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:19:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:19:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:19:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:19:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:19:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:19:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:19:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:19:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:19:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:19:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:19:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:19:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:19:05.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:05.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:05.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:05.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:19:05.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:19:05.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:19:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:19:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:19:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:19:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:19:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:19:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:19:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:19:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:19:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:19:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:19:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:19:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:19:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:19:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:19:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:19:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:19:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:19:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:19:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:19:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:19:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:19:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:19:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:19:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:19:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:19:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:19:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:19:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:19:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:19:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:19:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:19:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:19:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:19:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:19:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:19:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:19:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:19:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:19:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:19:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:19:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:19:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:19:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:19:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:19:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:19:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:19:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:19:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:19:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:19:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:19:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:19:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:19:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:19:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:19:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:19:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:19:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:19:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:19:07.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:19:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:19:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:19:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:19:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:19:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:19:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:19:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:19:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:19:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:19:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:19:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:19:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:19:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:19:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:19:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:19:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:19:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:19:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:19:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:19:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:19:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:19:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:19:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:19:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:19:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:19:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:19:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:19:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:19:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:19:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:19:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:19:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:19:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:19:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:19:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:19:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:19:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:19:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:19:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:19:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:05
2024-05-08T01:19:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:19:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:19:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:19:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:07'
2024-05-08T01:19:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:19:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:19:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:19:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:19:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:19:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:19:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:19:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:09
2024-05-08T01:19:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:19:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:19:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:19:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:07'
2024-05-08T01:19:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:19:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:19:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:19:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:19:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:19:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:19:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:19:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:19:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:19:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:19:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:19:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:19:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:09'
2024-05-08T01:19:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:19:09'
2024-05-08T01:19:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:19:09'
2024-05-08T01:19:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:19:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:19:09</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:19:07</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:19:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:19:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:19:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:09'
2024-05-08T01:19:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:19:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:19:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:19:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:19:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:19:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:19:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:19:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:19:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:19:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:19:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:07
2024-05-08T01:19:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:19:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:19:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:19:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:09'
2024-05-08T01:19:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:19:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:19:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:19:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:19:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:19:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:19:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:19:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:19:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:19:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:19:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:19:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:19:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:07'
2024-05-08T01:19:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:19:07'
2024-05-08T01:19:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:19:07'
2024-05-08T01:19:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:19:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:19:07</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:19:09</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:19:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:19:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:19:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:07'
2024-05-08T01:19:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:19:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:19:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:19:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:19:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:19:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:19:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:19:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:19:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:19:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:19:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:19:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:19:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:19:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:19:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:19:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:19:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:19:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:19:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:19:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:19:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:19:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:19:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:19:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:19:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:19:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:19:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:19:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:19:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:19:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:19:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:19:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:19:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:19:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:19:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:19:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:19:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:19:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:19:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:19:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:19:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:19:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:19:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:19:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:19:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:19:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:19:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:19:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:19:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:19:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:19:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:19:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:19:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:19:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:19:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:19:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:19:13.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:19:13.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:19:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:19:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:19:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:19:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:19:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:19:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:19:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:19:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:19:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:19:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:19:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:19:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:19:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:19:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:19:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:19:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:19:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:19:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:19:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:19:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:19:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:19:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:19:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:19:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:19:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:19:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:19:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:19:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:19:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:19:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:19:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:19:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:19:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:19:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:19:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:19:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:19:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:19:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:19:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:19:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:19:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:19:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:19:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:19:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:19:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:19:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:19:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:19:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:19:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:19:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:19:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:19:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:19:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:19:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:19:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:19:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:19:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:19:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:19:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:19:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:19:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:19:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:19:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:19:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:19:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:19:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:19:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=24, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:19:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=24, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:19:13.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:13.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=24, entry_tid=0'
2024-05-08T01:19:13.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:19:13.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:19:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:19:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:19:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:19:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:19:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:19:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:19:13.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:13.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:19:13.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:19:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:19:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:19:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=24, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:19:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=24, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:19:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=24, entry_tid=0'
2024-05-08T01:19:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=24, entry_tid=0' -a 'object_number=0, tag_tid=24, entry_tid=0' = 'object_number=0, tag_tid=24, entry_tid=0'
2024-05-08T01:19:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:19:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:19:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:14.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:19:14.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:19:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:19:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:19:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:19:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:14
2024-05-08T01:19:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:19:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:19:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":21},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":24}}
2024-05-08T01:19:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:12'
2024-05-08T01:19:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:19:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:19:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:19:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:19:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:19:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:19:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:19:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:19:14.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:19:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:19:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:19:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:19:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:14'
2024-05-08T01:19:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:19:14'
2024-05-08T01:19:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:19:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:19:14'
2024-05-08T01:19:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:19:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:19:14</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:21},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:24}}</description><last_update>2024-05-08 01:19:12</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:19:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:19:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:19:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:14'
2024-05-08T01:19:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:19:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:19:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:19:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:19:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:19:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:19:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:19:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:19:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:19:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:15.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:19:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:19:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":21},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":24}}
2024-05-08T01:19:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:12
2024-05-08T01:19:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:19:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:19:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:19:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:14'
2024-05-08T01:19:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:19:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:19:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:19:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":21},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":24}}
2024-05-08T01:19:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:19:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:19:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:19:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:19:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:19:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:19:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:19:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:19:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:12'
2024-05-08T01:19:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:19:12'
2024-05-08T01:19:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:19:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:19:12'
2024-05-08T01:19:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:19:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:21},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:24}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:19:12</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:19:14</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:19:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:19:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:19:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:12'
2024-05-08T01:19:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:19:15.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:19:15.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:19:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:19:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:19:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:19:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:19:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:19:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:19:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:19:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:19:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:19:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:19:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:19:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:19:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:19:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:19:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:19:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:19:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:19:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:19:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:19:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:19:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:19:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:19:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:19:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:19:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:19:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:19:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:19:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:19:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:19:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:19:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:19:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:19:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:19:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:19:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:19:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:19:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:19:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:19:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:19:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:19:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:19:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:19:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:19:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:19:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:19:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:19:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:19:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:19:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:19:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:19:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:19:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:19:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:19:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:19:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:19:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:19:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:19:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:19:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:19:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:19:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:19:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:19:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:19:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:19:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:19:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:19:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:19:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:19:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:19:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:19:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:19:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:19:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:19:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:19:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:19:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:19:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:19:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:19:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:19:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:19:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:19:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:19:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:19:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:19:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:19:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:19:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:19:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:19:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:19:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:19:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:19:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:19:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:19:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:19:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:19:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:19:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:19:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:19:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:19:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:19:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:19:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:19:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:19:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:19:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:19:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:19:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:19:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:19:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:19:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:19
2024-05-08T01:19:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:19:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:19:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:19:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:17'
2024-05-08T01:19:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:19:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:19:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:19:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:19:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:19:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:19:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:19:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:19:20.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:19:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:19:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:19:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:19:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:19'
2024-05-08T01:19:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:19:19'
2024-05-08T01:19:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:19:19'
2024-05-08T01:19:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:19:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:19:19</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>stopped</description><last_update>2024-05-08 01:19:17</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:19:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:19:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:19:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:19'
2024-05-08T01:19:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:19:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:19:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:19:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:19:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:19:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:19:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:19:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:21.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:21.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:19:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:19:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:19:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:17
2024-05-08T01:19:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:19:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:19:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:19:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:19'
2024-05-08T01:19:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:19:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:19:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:19:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:19:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:23.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:19:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:19:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:19:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:22
2024-05-08T01:19:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:19:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:19:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:19:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:19'
2024-05-08T01:19:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:19:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:19:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:19:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:19:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:19:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:19:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:19:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:19:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:19:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:19:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:19:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:19:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:22'
2024-05-08T01:19:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:19:22'
2024-05-08T01:19:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:19:22'
2024-05-08T01:19:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:19:23.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:19:22</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:19:19</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:19:23.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:19:23.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:19:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:22'
2024-05-08T01:19:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:19:23.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:19:23.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:19:23.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:19:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:19:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:19:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:19:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:19:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:19:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:19:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:19:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:19:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:19:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:19:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:19:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:19:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:19:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:19:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:19:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:19:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:19:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:19:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:19:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:19:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:19:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:19:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:19:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:19:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:19:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:19:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:19:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:19:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:19:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:19:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:19:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:19:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:19:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:19:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:19:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:19:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:19:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:19:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:19:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:19:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:19:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:19:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:19:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:19:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:19:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:19:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:19:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:19:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:19:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:19:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:19:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:19:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:19:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:19:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:19:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:19:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:19:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:19:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:19:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:19:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:19:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:19:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:19:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:19:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:19:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:19:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:19:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:19:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:19:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:19:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:19:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:19:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:19:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:19:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:19:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:19:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:19:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:19:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:19:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:19:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:19:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:19:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:19:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:19:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:19:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:19:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:19:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:19:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:19:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:19:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:19:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:19:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:19:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:19:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:19:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:19:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:19:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:19:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:19:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:19:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:19:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:19:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:19:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:19:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:19:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:19:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:19:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:19:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:19:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:19:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:19:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:19:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:19:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:19:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:19:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:19:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:19:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:19:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:19:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:19:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=28, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:19:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=28, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:19:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=28, entry_tid=0'
2024-05-08T01:19:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:19:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:19:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:19:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:19:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:19:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:19:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:19:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:19:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:19:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:19:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=28, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:19:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=28, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:19:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=28, entry_tid=0'
2024-05-08T01:19:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=28, entry_tid=0' -a 'object_number=0, tag_tid=28, entry_tid=0' = 'object_number=0, tag_tid=28, entry_tid=0'
2024-05-08T01:19:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:19:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:19:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:19:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:19:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:19:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:19:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:19:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:22
2024-05-08T01:19:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:19:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:19:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":25},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":28}}
2024-05-08T01:19:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:24'
2024-05-08T01:19:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:19:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:19:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:19:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:19:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:19:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:19:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:19:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:19:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:27
2024-05-08T01:19:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:19:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:19:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":25},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":28}}
2024-05-08T01:19:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:24'
2024-05-08T01:19:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:19:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:19:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:19:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:19:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:19:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:19:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:19:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:19:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:19:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:19:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:19:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:19:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:27'
2024-05-08T01:19:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:19:27'
2024-05-08T01:19:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:19:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:19:27'
2024-05-08T01:19:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:19:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:19:27</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:25},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:28}}</description><last_update>2024-05-08 01:19:24</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:19:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:19:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:19:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:27'
2024-05-08T01:19:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:19:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:19:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:19:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:19:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:19:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:19:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:19:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:19:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:19:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:19:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:19:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":25},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":28}}
2024-05-08T01:19:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:24
2024-05-08T01:19:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:19:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:19:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:19:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:27'
2024-05-08T01:19:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:19:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:19:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:19:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":25},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":28}}
2024-05-08T01:19:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:19:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:19:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:19:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:19:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:19:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:19:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:19:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:19:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:24'
2024-05-08T01:19:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:19:24'
2024-05-08T01:19:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:19:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:19:24'
2024-05-08T01:19:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:19:29.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:25},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:28}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:19:24</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:19:27</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:19:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:19:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:19:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:24'
2024-05-08T01:19:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:19:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:19:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:19:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:19:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:19:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:19:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:19:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:19:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:19:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:19:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:19:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:19:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:19:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:19:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:19:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:19:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:19:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:19:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:19:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:19:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:19:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:19:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:19:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:19:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:19:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:19:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:19:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:19:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:19:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:19:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:19:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:19:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:19:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:19:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:19:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:19:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:19:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:19:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:19:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:19:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:19:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:19:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:19:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:19:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:19:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:19:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:19:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:19:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:19:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:19:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:19:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:19:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:19:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:19:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:19:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:19:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:19:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:19:31.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:19:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:19:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:19:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:19:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:19:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:19:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:19:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:30
2024-05-08T01:19:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:19:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:19:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:19:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:32'
2024-05-08T01:19:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:19:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:19:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:19:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:19:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:19:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:19:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:19:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:30
2024-05-08T01:19:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:19:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:19:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:19:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:32'
2024-05-08T01:19:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:19:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:19:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 4
2024-05-08T01:19:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:19:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:19:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:19:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:19:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:34
2024-05-08T01:19:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:19:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:19:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:19:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:37'
2024-05-08T01:19:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:19:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:19:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:19:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:19:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:19:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:19:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:19:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:19:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:19:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:19:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:19:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:19:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:34'
2024-05-08T01:19:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:19:34'
2024-05-08T01:19:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:19:34'
2024-05-08T01:19:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:19:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:19:34</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:19:37</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:19:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:19:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:19:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:34'
2024-05-08T01:19:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:19:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:19:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:19:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:19:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:19:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:19:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:19:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:19:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:19:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:19:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:37
2024-05-08T01:19:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:19:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:19:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:19:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:39'
2024-05-08T01:19:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:19:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:19:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:19:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:19:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:19:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:19:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:19:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:19:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:19:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:19:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:19:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:19:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:37'
2024-05-08T01:19:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:19:37'
2024-05-08T01:19:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:19:37'
2024-05-08T01:19:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:19:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:19:37</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:19:39</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:19:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:19:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:19:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:37'
2024-05-08T01:19:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:19:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:19:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:19:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:19:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:19:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:19:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:19:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:19:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:19:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:19:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:19:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:19:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:19:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:19:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:19:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:19:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:19:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:19:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:19:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:19:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:19:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:19:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:19:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:19:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:19:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:19:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:19:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:19:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:19:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:19:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:19:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:19:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:19:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:19:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:19:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:19:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:19:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:19:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:19:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:19:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:19:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:19:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:19:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:19:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:19:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:19:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:19:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:19:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:19:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:19:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:19:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:19:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:19:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:19:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:19:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:19:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:19:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:19:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:19:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:19:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:19:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:19:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:19:42.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:19:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:19:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:19:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:19:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:19:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:19:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:19:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:19:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:19:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:19:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:19:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:19:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:19:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:19:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:19:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:19:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:19:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:19:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:19:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:19:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:19:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:19:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:19:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:19:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:19:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:19:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:19:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:19:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:19:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:19:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:19:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:19:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:19:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:19:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:19:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:19:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:19:42.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:19:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:19:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:19:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:19:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:19:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:19:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:19:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:19:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:19:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:19:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:19:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:19:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:19:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:19:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:19:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:19:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:19:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:19:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:19:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:19:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:19:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:19:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:19:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:19:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:19:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:19:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:19:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:19:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:19:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:19:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:19:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:19:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:19:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:19:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:19:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:19:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:19:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:19:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:19:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:19:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:19:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:19:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:19:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:19:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:19:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:19:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:19:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:19:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:19:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:19:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:19:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:19:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:19:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:19:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:19:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:19:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:19:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:19:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:19:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:19:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:19:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:19:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:19:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:19:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:19:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:19:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:19:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:19:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:19:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:19:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:19:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=29, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:19:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=29, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:19:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=29, entry_tid=0'
2024-05-08T01:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:19:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:19:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:19:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:19:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:19:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:19:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:19:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:19:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=29, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:19:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=29, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:19:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:43.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=29, entry_tid=0'
2024-05-08T01:19:43.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=29, entry_tid=0' -a 'object_number=0, tag_tid=29, entry_tid=0' = 'object_number=0, tag_tid=29, entry_tid=0'
2024-05-08T01:19:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:19:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:19:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:19:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:19:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:19:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:19:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:19:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:44
2024-05-08T01:19:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:19:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:19:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":26},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":29}}
2024-05-08T01:19:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:42'
2024-05-08T01:19:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:19:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:19:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:19:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:19:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:19:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:19:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:19:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:19:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:19:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:19:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:19:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:19:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:44'
2024-05-08T01:19:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:19:44'
2024-05-08T01:19:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:19:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:19:44'
2024-05-08T01:19:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:19:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:19:44</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:26},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:29}}</description><last_update>2024-05-08 01:19:42</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:19:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:19:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:19:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:44'
2024-05-08T01:19:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:19:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:19:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:19:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:19:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:19:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:19:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:19:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:19:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:19:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:19:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:19:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":26},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":29}}
2024-05-08T01:19:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:42
2024-05-08T01:19:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:19:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:19:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:19:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:44'
2024-05-08T01:19:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:19:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:19:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:19:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":26},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":29}}
2024-05-08T01:19:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:19:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:19:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:19:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:19:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:19:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:19:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:19:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:19:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:42'
2024-05-08T01:19:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:19:42'
2024-05-08T01:19:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:19:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:19:42'
2024-05-08T01:19:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:19:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:26},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:29}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:19:42</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:19:44</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:19:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:19:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:19:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:42'
2024-05-08T01:19:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:19:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:19:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:19:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:19:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:19:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:19:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:19:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:19:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:19:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:19:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:19:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:19:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:19:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:19:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:19:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:19:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:19:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:19:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:19:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:19:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:19:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:19:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:19:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:19:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:19:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:19:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:19:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:19:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:19:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:19:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:19:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:19:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:19:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:19:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:19:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:19:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:19:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:19:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:19:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:19:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:19:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:19:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:19:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:19:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:19:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:19:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:19:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:19:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:19:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:19:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:19:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:19:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:19:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:19:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:19:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:19:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:19:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:19:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:19:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:19:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:19:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:19:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:19:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:19:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:19:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:19:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:19:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:19:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:19:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:19:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:19:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:19:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:19:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:19:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:19:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:19:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:19:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:19:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:19:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:19:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:19:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:19:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:19:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:19:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:19:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:19:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:19:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:19:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:19:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:19:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:19:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:19:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:19:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:19:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:19:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:19:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:19:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:19:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:19:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:19:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:19:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:19:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:19:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:19:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:50.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:50.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:19:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:19:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:19:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:49
2024-05-08T01:19:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:19:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:19:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:19:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:47'
2024-05-08T01:19:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:19:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:19:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:19:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:19:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:19:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:19:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:19:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:19:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:19:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:19:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:19:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:19:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:49'
2024-05-08T01:19:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:19:49'
2024-05-08T01:19:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:19:49'
2024-05-08T01:19:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:19:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:19:49</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>stopped</description><last_update>2024-05-08 01:19:47</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:19:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:19:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:19:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:49'
2024-05-08T01:19:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:19:50.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:19:50.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:19:50.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:19:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:19:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:19:50.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:50.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:50.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:50.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:50.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:50.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:50.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:50.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:51.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:19:51.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:51.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:51.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:51.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:51.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:19:51.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:19:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:19:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:47
2024-05-08T01:19:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:19:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:19:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:19:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:49'
2024-05-08T01:19:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:19:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:19:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:19:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:19:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:19:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:19:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:19:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:52
2024-05-08T01:19:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:19:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:19:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:19:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:49'
2024-05-08T01:19:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:19:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:19:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:19:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:19:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:19:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:19:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:19:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:19:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:19:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:19:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:19:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:19:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:52'
2024-05-08T01:19:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:19:52'
2024-05-08T01:19:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:19:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:19:52'
2024-05-08T01:19:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:19:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:19:52</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:19:49</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:19:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:19:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:19:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:52'
2024-05-08T01:19:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:19:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:19:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:19:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:19:53.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:19:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:19:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:19:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:19:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:19:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:19:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:19:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:19:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:19:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:19:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:19:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:19:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:19:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:19:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:19:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:19:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:19:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:19:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:19:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:19:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:19:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:19:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:19:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:19:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:19:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:19:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:19:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:19:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:19:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:19:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:19:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:19:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:19:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:19:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:19:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:19:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:19:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:19:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:19:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:19:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:19:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:19:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:19:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:19:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:19:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:19:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:19:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:19:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:19:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:19:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:19:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:19:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:19:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:19:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:19:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:19:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:19:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:19:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:19:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:19:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:19:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:19:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:19:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:19:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:19:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:19:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:19:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:19:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:19:55.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:19:55.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:55.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:55.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:55.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:19:55.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:19:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:19:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:19:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:19:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:19:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:19:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:19:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:19:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:19:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:19:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:19:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:19:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:19:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:19:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:19:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:19:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:19:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:19:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:19:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:19:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:19:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:19:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:19:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:19:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:19:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:19:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:19:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:19:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:19:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:19:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:19:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:19:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:19:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:19:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:19:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:19:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:19:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:19:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:19:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:19:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:19:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:19:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:19:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:19:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:19:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:19:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:19:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:19:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:19:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:19:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:19:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:19:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:19:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=33, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:19:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=33, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:19:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=33, entry_tid=0'
2024-05-08T01:19:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:19:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:19:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:19:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:19:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:19:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:19:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:19:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:19:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:19:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:19:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:55.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:19:55.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:19:55.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:19:55.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=33, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:19:55.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=33, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:19:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:19:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=33, entry_tid=0'
2024-05-08T01:19:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=33, entry_tid=0' -a 'object_number=0, tag_tid=33, entry_tid=0' = 'object_number=0, tag_tid=33, entry_tid=0'
2024-05-08T01:19:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:19:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:19:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:19:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:19:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:19:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:19:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:19:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:52
2024-05-08T01:19:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:19:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:19:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":30},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":33}}
2024-05-08T01:19:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:54'
2024-05-08T01:19:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:19:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:19:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:19:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:19:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:19:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:19:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:19:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:19:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:57
2024-05-08T01:19:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:19:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:19:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":30},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":33}}
2024-05-08T01:19:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:54'
2024-05-08T01:19:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:19:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:19:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:19:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:19:58.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:19:58.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:19:58.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:19:58.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:19:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:19:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:19:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:19:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:19:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:57'
2024-05-08T01:19:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:19:57'
2024-05-08T01:19:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:19:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:19:57'
2024-05-08T01:19:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:19:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:19:57</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:30},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:33}}</description><last_update>2024-05-08 01:19:54</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:19:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:19:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:19:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:57'
2024-05-08T01:19:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:19:58.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:19:58.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:19:58.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:19:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:19:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:19:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:19:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:19:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:19:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:19:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:19:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:19:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:19:59.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:19:59.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:19:59.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:19:59.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":30},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":33}}
2024-05-08T01:19:59.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:19:59.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:19:54
2024-05-08T01:19:59.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:19:59.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:19:59.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:19:59.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:19:59.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:19:57'
2024-05-08T01:19:59.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:19:59.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:19:59.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:19:59.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":30},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":33}}
2024-05-08T01:19:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:19:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:19:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:19:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:19:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:19:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:19:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:19:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:19:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:54'
2024-05-08T01:19:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:19:54'
2024-05-08T01:19:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:19:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:19:54'
2024-05-08T01:19:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:19:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:19:59.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:19:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:30},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:33}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:19:54</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:19:57</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:19:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:19:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:19:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:19:54'
2024-05-08T01:19:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:19:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:19:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:19:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:19:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:19:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:19:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:19:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:19:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:19:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:19:59.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:19:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:19:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:19:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:19:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:19:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:19:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:19:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:19:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:19:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:20:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:20:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:20:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:20:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:20:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:20:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:20:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:20:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:20:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:20:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:20:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:20:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:20:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:20:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:20:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:20:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:20:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:20:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:20:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:20:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:20:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:20:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:20:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:20:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:20:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:20:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:20:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:20:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:20:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:20:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:20:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:20:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:20:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:20:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:20:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:20:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:20:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:20:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:20:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:20:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:20:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:20:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:20:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:20:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:20:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:20:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:20:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:20:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:20:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:20:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:20:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:20:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:20:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:20:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:20:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:20:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:20:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:20:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:20:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:20:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:20:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:20:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:20:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:20:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:20:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:20:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:20:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:20:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:20:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:20:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:20:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:20:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:20:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:20:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:20:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:20:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:20:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:20:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:20:02.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:02.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:02.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:02.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:20:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:20:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:20:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:20:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:20:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:20:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:20:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:20:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:20:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:20:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:20:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:00
2024-05-08T01:20:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:20:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:20:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:20:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:02'
2024-05-08T01:20:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:20:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:20:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:20:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:20:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:20:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:20:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:20:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:04
2024-05-08T01:20:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:20:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:20:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:20:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:02'
2024-05-08T01:20:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:20:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:20:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:20:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:20:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:20:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:20:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:20:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:20:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:20:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:20:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:20:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:20:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:04'
2024-05-08T01:20:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:20:04'
2024-05-08T01:20:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:20:04'
2024-05-08T01:20:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:20:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:20:04</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:20:02</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:20:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:20:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:20:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:04'
2024-05-08T01:20:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:20:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:20:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:20:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:20:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:20:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:20:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:07.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:20:07.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:07.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:07.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:07.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:07.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:07.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:07.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:07.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:07.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:20:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:20:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:20:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:07
2024-05-08T01:20:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:20:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:20:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:20:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:04'
2024-05-08T01:20:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:20:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:20:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:20:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:20:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:20:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:20:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:20:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:20:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:20:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:20:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:20:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:20:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:07'
2024-05-08T01:20:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:20:07'
2024-05-08T01:20:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:20:07'
2024-05-08T01:20:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:20:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:20:07</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:20:04</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:20:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:20:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:20:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:07'
2024-05-08T01:20:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:20:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:20:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:20:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:20:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:20:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:20:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:20:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:20:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:20:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:20:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:20:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:20:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:20:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:20:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:20:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:20:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:20:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:20:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:20:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:20:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:20:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:20:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:20:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:20:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:20:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:20:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:20:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:20:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:20:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:20:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:20:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:20:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:20:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:20:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:20:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:20:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:20:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:20:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:20:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:20:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:20:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:20:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:20:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:20:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:20:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:20:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:20:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:20:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:20:10.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:20:10.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:20:10.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:20:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:20:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:20:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:20:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:20:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:20:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:20:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:20:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:20:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:20:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:20:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:20:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:20:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:20:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:20:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:20:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:20:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:20:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:20:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:20:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:20:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:20:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:20:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:20:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:20:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:20:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:20:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:20:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:20:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:20:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:20:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:20:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:20:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:20:10.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:20:10.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:10.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 4
2024-05-08T01:20:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:20:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:20:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:20:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:20:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:20:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:20:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:20:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:20:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:20:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:20:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:20:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:20:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:20:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:20:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:20:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:20:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:20:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:20:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:20:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:20:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:20:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:20:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:20:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:20:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:20:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:20:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:20:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:20:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:20:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:20:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:20:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:20:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:20:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:20:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:20:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:20:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:20:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:20:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:20:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:20:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:20:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:20:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:20:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:20:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:20:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:20:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:20:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:20:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:20:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:20:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:20:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:20:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:20:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:20:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:20:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:20:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:20:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:20:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:20:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:20:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:20:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:20:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:20:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:20:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:20:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:20:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:20:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:20:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:20:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:20:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:20:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:20:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:20:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:20:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:20:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:20:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:20:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:20:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:20:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:20:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:20:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:20:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:20:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:20:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:20:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:20:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:20:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:20:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:20:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:20:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:20:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:20:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:20:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:20:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:20:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:20:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:20:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:20:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:20:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:20:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:20:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:20:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:20:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:20:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:20:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=34, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:20:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=34, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:20:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=34, entry_tid=0'
2024-05-08T01:20:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:20:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:20:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:20:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:20:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:20:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:20:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:20:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:20:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:20:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:20:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:20:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:20:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:20:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:20:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:20:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:20:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=34, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:20:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=34, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:20:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=34, entry_tid=0'
2024-05-08T01:20:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=34, entry_tid=0' -a 'object_number=0, tag_tid=34, entry_tid=0' = 'object_number=0, tag_tid=34, entry_tid=0'
2024-05-08T01:20:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:20:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:20:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:20:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:20:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:20:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:20:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:20:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:14
2024-05-08T01:20:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:20:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:20:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":31},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":34}}
2024-05-08T01:20:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:12'
2024-05-08T01:20:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:20:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:20:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:20:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:20:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:20:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:20:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:20:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:20:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:20:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:20:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:20:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:20:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:14'
2024-05-08T01:20:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:20:14'
2024-05-08T01:20:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:20:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:20:14'
2024-05-08T01:20:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:20:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:20:14</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:31},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:34}}</description><last_update>2024-05-08 01:20:12</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:20:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:20:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:20:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:14'
2024-05-08T01:20:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:20:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:20:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:20:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:20:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:20:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:20:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:20:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:20:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:20:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:20:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:20:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":31},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":34}}
2024-05-08T01:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:12
2024-05-08T01:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:14'
2024-05-08T01:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:20:17.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:20:17.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":31},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":34}}
2024-05-08T01:20:17.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:20:17.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:20:17.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:20:17.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:20:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:20:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:20:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:20:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:20:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:12'
2024-05-08T01:20:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:20:12'
2024-05-08T01:20:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:20:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:20:12'
2024-05-08T01:20:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:20:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:31},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:34}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:20:12</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:20:14</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:20:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:20:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:20:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:12'
2024-05-08T01:20:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:20:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:20:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:20:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:20:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:20:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:20:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:20:18.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:20:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:20:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:20:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:20:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:20:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:20:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:20:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:20:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:20:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:20:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:20:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:20:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:20:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:20:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:20:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:20:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:20:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:20:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:20:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:20:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:20:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:20:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:20:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:20:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:20:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:20:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:20:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:20:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:20:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:20:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:20:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:20:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:20:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:20:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:20:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:20:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:20:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:20:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:20:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:20:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:20:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:20:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:20:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:20:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:20:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:20:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:20:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:20:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:20:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:20:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:20:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:20:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:20:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:20:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:20:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:20:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:20:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:20:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:20:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:20:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:20:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:20:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:20:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:20:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:20:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:20:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:20:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:20:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:20:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:20:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:20:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:20:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:20:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:20:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:20:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:20:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:20:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:20:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:20:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:20:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:20:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:20:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:20:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:20:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:20:21.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:20:21.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:20:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:20:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:20:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:20:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:20:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:20:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:20:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:20:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:20:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:19
2024-05-08T01:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:22'
2024-05-08T01:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:20:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:20:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:20:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:20:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:20:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:20:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:20:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:20:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:20:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:20:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:20:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:19'
2024-05-08T01:20:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:20:19'
2024-05-08T01:20:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:20:19'
2024-05-08T01:20:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:20:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:20:19</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:20:22</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:20:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:20:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:20:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:19'
2024-05-08T01:20:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:20:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:20:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:20:22.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:20:22.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:20:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:20:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:20:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:20:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:20:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:20:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:22
2024-05-08T01:20:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:20:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:20:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:20:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:19'
2024-05-08T01:20:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:20:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:20:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:20:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:20:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:20:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:20:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:20:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:20:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:20:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:20:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:20:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:20:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:22'
2024-05-08T01:20:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:20:22'
2024-05-08T01:20:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:20:22'
2024-05-08T01:20:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:20:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:20:22</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:20:19</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:20:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:20:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:20:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:22'
2024-05-08T01:20:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:20:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:20:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:20:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:20:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:20:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:20:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:20:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:20:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:20:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:20:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:20:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:20:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:20:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:20:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:20:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:20:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:20:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:20:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:20:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:20:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:20:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:20:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:20:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:20:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:20:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:20:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:20:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:20:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:20:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:20:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:20:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:20:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:20:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:20:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:20:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:20:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:20:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:20:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:20:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:20:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:20:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:20:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:20:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:20:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:20:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:20:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:20:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:20:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:20:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:20:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:20:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:20:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:20:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:20:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:20:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:20:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:20:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:20:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:20:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:20:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:20:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:20:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:20:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:20:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:20:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:20:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:20:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:20:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:20:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:20:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:20:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:20:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:20:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:20:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:20:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:20:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:20:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:20:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:20:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:20:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:20:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:20:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:20:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:20:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:20:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:20:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:20:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:20:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:20:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:20:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:20:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:20:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:20:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:20:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:20:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:20:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:20:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:20:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:20:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:20:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:20:25.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:25.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:25.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:25.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:20:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:20:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:20:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:20:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:20:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:20:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:20:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:20:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:20:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:20:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:20:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:20:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:20:25.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:25.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:20:25.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:20:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:20:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:20:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=38, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:20:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=38, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:20:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=38, entry_tid=0'
2024-05-08T01:20:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:20:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:20:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:20:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:20:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:20:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:20:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:20:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:20:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:20:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:20:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:20:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:20:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:20:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:20:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:20:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:20:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=38, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:20:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=38, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:20:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=38, entry_tid=0'
2024-05-08T01:20:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=38, entry_tid=0' -a 'object_number=0, tag_tid=38, entry_tid=0' = 'object_number=0, tag_tid=38, entry_tid=0'
2024-05-08T01:20:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:20:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:20:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:20:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:20:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:20:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:20:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:20:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:22
2024-05-08T01:20:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:20:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:20:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":35},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38}}
2024-05-08T01:20:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:24'
2024-05-08T01:20:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:20:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:20:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:20:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:20:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:20:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:20:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:20:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:20:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:27
2024-05-08T01:20:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:20:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:20:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":35},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38}}
2024-05-08T01:20:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:24'
2024-05-08T01:20:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:20:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:20:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:20:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:20:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:20:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:20:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:20:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:20:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:20:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:20:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:20:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:20:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:27'
2024-05-08T01:20:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:20:27'
2024-05-08T01:20:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:20:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:20:27'
2024-05-08T01:20:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:20:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:20:27</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:35},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:38}}</description><last_update>2024-05-08 01:20:24</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:20:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:20:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:20:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:27'
2024-05-08T01:20:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:20:28.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:20:28.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:20:28.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:20:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:20:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:20:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:20:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:20:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:20:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:20:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:20:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":35},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38}}
2024-05-08T01:20:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:24
2024-05-08T01:20:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:20:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:20:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:20:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:27'
2024-05-08T01:20:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:20:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:20:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:20:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":35},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38}}
2024-05-08T01:20:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:20:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:20:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:20:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:20:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:20:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:20:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:20:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:20:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:24'
2024-05-08T01:20:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:20:24'
2024-05-08T01:20:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:20:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:20:24'
2024-05-08T01:20:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:20:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:35},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:38}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:20:24</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:20:27</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:20:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:20:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:20:29.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:24'
2024-05-08T01:20:29.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:20:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:20:29.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:20:29.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:20:29.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:20:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:20:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:20:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:20:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:20:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:20:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:20:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:20:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:20:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:20:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:20:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:20:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:20:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:20:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:20:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:20:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:20:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:20:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:20:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:20:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:20:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:20:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:20:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:20:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:20:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:20:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:20:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:20:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:20:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:20:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:20:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:20:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:20:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:20:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:20:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:20:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:20:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:20:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:20:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:20:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:20:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:20:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:20:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:20:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:20:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:20:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:20:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:20:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:20:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:20:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:20:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:20:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:20:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:20:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:20:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:20:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:20:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:20:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:20:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:20:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:20:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:20:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:20:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:20:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:20:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:20:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:20:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:20:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:20:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:20:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:20:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:20:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:20:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:20:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:20:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:20:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:20:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:20:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:20:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:20:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:20:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:20:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:20:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:20:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:20:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:20:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:20:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:20:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:20:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:20:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:20:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:20:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:20:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:20:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:20:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:20:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:20:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:20:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:20:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:34
2024-05-08T01:20:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:20:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:20:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:20:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:32'
2024-05-08T01:20:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:20:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:20:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:20:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:20:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:20:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:20:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:20:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:20:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:20:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:20:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:20:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:20:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:34'
2024-05-08T01:20:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:20:34'
2024-05-08T01:20:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:20:34'
2024-05-08T01:20:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:20:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:20:34</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:20:32</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:20:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:20:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:20:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:34'
2024-05-08T01:20:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:20:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:20:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:20:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:20:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:20:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:20:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:20:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:20:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:20:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:20:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:32
2024-05-08T01:20:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:20:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:20:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:20:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:34'
2024-05-08T01:20:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:20:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:20:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:20:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:20:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:20:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:20:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:20:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:20:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:20:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:20:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:20:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:20:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:32'
2024-05-08T01:20:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:20:32'
2024-05-08T01:20:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:20:32'
2024-05-08T01:20:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:20:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:20:32</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:20:34</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:20:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:20:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:20:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:32'
2024-05-08T01:20:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:20:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:20:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:20:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:20:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:20:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:20:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:20:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:20:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:20:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:20:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:20:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:20:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:20:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:20:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:20:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:20:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:20:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:20:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:20:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:20:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:20:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:20:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:20:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:20:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:20:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:20:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:20:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:20:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:20:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:20:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:20:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:20:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:20:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:20:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:20:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:20:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:20:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:20:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:20:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:20:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:20:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:20:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:20:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:20:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:20:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:20:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:20:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:20:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:20:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:20:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:20:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:20:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:20:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:20:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:20:38.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:20:38.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:38.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:38.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:20:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:20:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:20:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:20:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:20:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:20:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:20:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:20:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:20:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:20:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:20:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:20:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:20:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:20:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:20:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:20:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:20:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:20:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:20:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:20:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:20:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:20:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:20:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:20:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:20:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:20:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:20:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:20:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:20:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:20:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:20:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:20:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:20:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:20:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:38.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:38.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:38.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:20:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:20:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:20:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:20:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:20:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:20:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:20:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:20:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:20:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:20:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:20:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:20:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:20:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:20:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:20:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:20:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:20:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:20:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:20:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:20:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:20:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:20:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:20:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:20:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:20:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:20:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:20:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:20:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:20:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:20:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:20:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:20:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:20:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:20:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:20:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:20:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:20:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:20:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:20:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:20:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:20:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:20:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:20:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:20:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:20:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:20:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:20:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:20:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:20:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:20:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:20:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:20:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:20:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:20:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:20:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:20:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:20:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:20:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:20:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:20:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:20:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:20:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:20:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:20:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:20:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:20:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:20:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:20:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:20:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:20:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=39, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:20:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=39, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:20:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=39, entry_tid=0'
2024-05-08T01:20:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:20:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:20:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:20:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:20:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:20:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:20:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:20:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:20:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:20:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:20:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:20:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:20:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:20:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:20:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:20:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:20:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=39, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:20:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=39, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:20:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=39, entry_tid=0'
2024-05-08T01:20:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=39, entry_tid=0' -a 'object_number=0, tag_tid=39, entry_tid=0' = 'object_number=0, tag_tid=39, entry_tid=0'
2024-05-08T01:20:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:20:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:20:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:20:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:20:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:20:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:20:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:20:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:39
2024-05-08T01:20:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:20:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:20:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":36},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":39}}
2024-05-08T01:20:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:37'
2024-05-08T01:20:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:20:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:20:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:20:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:20:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:20:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:20:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:20:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:20:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:20:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:20:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:20:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:20:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:39'
2024-05-08T01:20:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:20:39'
2024-05-08T01:20:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:20:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:20:39'
2024-05-08T01:20:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:20:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:20:39</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:36},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:39}}</description><last_update>2024-05-08 01:20:37</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:20:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:20:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:20:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:39'
2024-05-08T01:20:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:20:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:20:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:20:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:20:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:20:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:20:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:20:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:20:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:20:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:20:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:20:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":36},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":39}}
2024-05-08T01:20:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:37
2024-05-08T01:20:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:20:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:20:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:20:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:39'
2024-05-08T01:20:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:20:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:20:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:20:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":36},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":39}}
2024-05-08T01:20:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:20:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:20:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:20:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:20:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:20:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:20:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:20:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:20:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:37'
2024-05-08T01:20:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:20:37'
2024-05-08T01:20:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:20:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:20:37'
2024-05-08T01:20:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:20:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:36},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:39}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:20:37</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:20:39</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:20:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:20:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:20:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:37'
2024-05-08T01:20:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:20:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:20:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:20:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:20:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:20:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:20:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:20:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:20:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:20:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:20:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:20:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:20:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:20:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:20:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:20:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:20:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:20:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:20:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:20:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:20:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:20:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:20:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:20:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:20:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:20:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:20:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:20:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:20:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:20:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:20:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:20:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:20:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:20:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:20:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:20:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:20:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:20:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:20:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:20:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:20:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:20:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:20:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:20:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:20:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:20:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:20:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:20:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:20:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:20:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:20:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:20:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:20:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:20:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:20:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:20:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:20:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:20:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:20:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:20:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:20:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:20:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:20:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:20:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:20:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:20:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:20:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:20:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:20:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:20:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:20:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:20:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:20:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:20:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:20:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:20:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:20:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:20:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:20:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:20:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:20:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:20:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:20:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:20:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:20:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:20:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:20:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:20:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:20:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:20:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:20:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:20:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:20:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:20:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:20:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:20:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:20:45.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:20:45.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:20:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:20:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:20:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:20:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:20:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:20:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:20:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:44
2024-05-08T01:20:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:20:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:20:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:20:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:43'
2024-05-08T01:20:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:20:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:20:46.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:20:46.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:20:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:20:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:20:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:20:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:20:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:20:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:20:46.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:20:46.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:20:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:44'
2024-05-08T01:20:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:20:44'
2024-05-08T01:20:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:20:44'
2024-05-08T01:20:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:20:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:20:44</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>stopped</description><last_update>2024-05-08 01:20:43</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:20:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:20:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:20:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:44'
2024-05-08T01:20:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:20:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:20:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:20:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:20:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:20:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:20:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:20:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:47.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:47.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:47.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:47.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:47.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:47.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:47.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:20:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:20:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:20:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:47
2024-05-08T01:20:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:20:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:20:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:20:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:44'
2024-05-08T01:20:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:20:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:20:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:20:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:20:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:20:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:20:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:20:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:20:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:20:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:20:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:20:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:20:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:47'
2024-05-08T01:20:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:20:47'
2024-05-08T01:20:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:20:47'
2024-05-08T01:20:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:20:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:20:47</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:20:44</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:20:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:20:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:20:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:47'
2024-05-08T01:20:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:20:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:20:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:20:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:20:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:20:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:20:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:20:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:20:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:20:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:20:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:20:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:20:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:20:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:20:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:20:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:20:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:20:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:20:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:20:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:20:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:20:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:20:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:20:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:20:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:20:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:20:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:20:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:20:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:20:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:20:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:20:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:20:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:20:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:20:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:20:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:20:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:20:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:20:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:20:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:20:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:20:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:20:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:20:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:20:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:20:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:20:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:20:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:20:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:20:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:20:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:20:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:20:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:20:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:20:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:20:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:20:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:20:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:20:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:20:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:20:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:20:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:20:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:20:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:20:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:20:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:20:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:20:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:20:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:20:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:20:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:20:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:20:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:20:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:20:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:20:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:20:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:20:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:20:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:20:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:20:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:20:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:20:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:20:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:20:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:20:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:20:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:20:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:20:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:20:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:20:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:20:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:20:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:20:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:20:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:20:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:20:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:20:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:20:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:20:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:20:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:20:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:20:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:20:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:20:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:20:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:20:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:20:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:20:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:20:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:20:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:20:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:20:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:20:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:20:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:20:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:20:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:20:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:20:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:20:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:20:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:20:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:20:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:20:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:20:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:20:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:20:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:20:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:20:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:20:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:20:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:20:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:20:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:20:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:20:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:20:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:20:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:20:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:20:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:20:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:20:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:20:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:20:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:20:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:20:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:20:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:20:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:20:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:20:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:20:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:20:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:20:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:20:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:20:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:20:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:20:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:20:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:20:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:20:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:20:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:20:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=43, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:20:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=43, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:20:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=43, entry_tid=0'
2024-05-08T01:20:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:20:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:20:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:20:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:20:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:20:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:20:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:20:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:20:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:20:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:20:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:20:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:20:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:20:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:20:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:20:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:20:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=43, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:20:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=43, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:20:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:20:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=43, entry_tid=0'
2024-05-08T01:20:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=43, entry_tid=0' -a 'object_number=0, tag_tid=43, entry_tid=0' = 'object_number=0, tag_tid=43, entry_tid=0'
2024-05-08T01:20:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:20:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:20:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:20:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:20:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:20:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:20:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:20:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:52
2024-05-08T01:20:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:20:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:20:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":40},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43}}
2024-05-08T01:20:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:49'
2024-05-08T01:20:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:20:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:20:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:20:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:20:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:20:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:20:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:20:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:20:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:20:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:20:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:20:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:20:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:52'
2024-05-08T01:20:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:20:52'
2024-05-08T01:20:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:20:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:20:52'
2024-05-08T01:20:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:20:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:20:52</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:40},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:43}}</description><last_update>2024-05-08 01:20:49</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:20:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:20:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:20:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:52'
2024-05-08T01:20:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:20:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:20:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:20:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:20:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:20:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:20:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:20:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:20:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:20:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:20:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:20:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":40},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43}}
2024-05-08T01:20:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:49
2024-05-08T01:20:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:20:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:20:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:20:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:52'
2024-05-08T01:20:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:20:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:20:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:20:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":40},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43}}
2024-05-08T01:20:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:20:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:20:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:20:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:20:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:20:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:20:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:20:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:20:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:49'
2024-05-08T01:20:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:20:49'
2024-05-08T01:20:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:20:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:20:49'
2024-05-08T01:20:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:20:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:40},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:43}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:20:49</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:20:52</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:20:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:20:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:20:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:49'
2024-05-08T01:20:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:20:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:20:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:20:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:20:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:20:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:20:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:20:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:20:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:20:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:20:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:20:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:20:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:20:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:20:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:20:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:20:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:20:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:20:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:20:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:20:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:20:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:20:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:20:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:20:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:20:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:20:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:20:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:20:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:20:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:20:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:20:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:20:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:20:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:20:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:20:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:20:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:20:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:20:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:20:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:20:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:20:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:20:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:20:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:20:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:20:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:20:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:20:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:20:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:20:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:20:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:20:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:20:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:20:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:20:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:20:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:20:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:20:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:20:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:20:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:20:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:20:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:20:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:20:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:20:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:20:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:20:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:20:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:20:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:20:56.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:56.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:56.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:20:56.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:20:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:55
2024-05-08T01:20:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:20:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:20:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:20:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:52'
2024-05-08T01:20:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:20:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:20:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:20:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:20:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:20:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:20:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:20:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:20:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:20:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:20:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:20:58.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:20:58.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:20:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:20:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:20:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:20:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:20:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:20:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:55
2024-05-08T01:20:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:20:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:20:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:20:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:20:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:57'
2024-05-08T01:20:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:20:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:20:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:20:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 4
2024-05-08T01:21:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:21:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:21:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:21:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:21:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:20:59
2024-05-08T01:21:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:21:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:21:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:21:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:02'
2024-05-08T01:21:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:21:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:21:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:21:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:21:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:21:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:21:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:21:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:21:02.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:21:02.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:21:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:59'
2024-05-08T01:21:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:20:59'
2024-05-08T01:21:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:20:59'
2024-05-08T01:21:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:21:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:20:59</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:21:02</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:21:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:21:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:21:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:20:59'
2024-05-08T01:21:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:21:02.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:21:02.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:21:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:21:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:21:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:21:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:21:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:21:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:02
2024-05-08T01:21:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:21:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:21:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:21:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:20:59'
2024-05-08T01:21:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:21:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:21:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:21:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:21:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:21:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:21:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:21:03.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:21:03.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:21:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:21:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:02'
2024-05-08T01:21:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:21:02'
2024-05-08T01:21:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:21:02'
2024-05-08T01:21:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:21:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:21:02</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:20:59</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:21:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:21:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:21:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:02'
2024-05-08T01:21:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:21:03.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:21:03.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:03.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:21:03.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:21:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:21:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:21:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:21:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:21:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:21:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:21:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:21:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:21:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:21:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:21:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:21:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:21:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:21:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:21:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:21:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:21:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:21:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:21:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:21:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:21:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:21:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:21:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:21:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:21:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:21:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:21:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:21:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:21:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:21:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:21:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:21:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:21:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:21:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:21:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:21:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:21:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:21:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:21:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:21:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:21:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:21:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:21:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:21:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:21:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:21:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:21:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:21:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:21:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:21:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:21:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:21:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:21:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:21:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:21:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:21:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:21:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:21:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:21:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:21:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:21:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:21:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:21:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:21:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:21:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:21:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:21:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:21:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:21:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:21:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:21:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:21:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:21:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:21:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:21:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:21:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:21:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:21:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:21:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:21:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:21:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:21:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:21:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:21:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:21:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:21:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:21:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:21:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:21:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:21:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:21:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:21:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:21:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:21:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:21:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:21:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:21:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:21:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:21:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:21:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:21:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:21:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:21:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:21:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:21:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:21:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:21:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:21:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:21:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:21:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:21:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:21:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:21:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:21:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:21:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:21:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:21:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:21:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:21:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:21:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:21:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:21:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:21:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:21:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:21:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:21:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:21:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:21:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:21:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:21:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:21:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:21:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:21:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:21:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:21:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:21:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:21:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:21:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:21:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:21:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:21:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:21:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:21:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:21:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:21:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:21:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=44, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:21:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=41, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:21:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=44, entry_tid=0'
2024-05-08T01:21:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:21:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:21:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:21:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:21:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:21:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:21:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:21:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:21:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:21:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:21:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=44, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:21:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=41, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:21:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=41, entry_tid=0'
2024-05-08T01:21:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=44, entry_tid=0' -a 'object_number=0, tag_tid=44, entry_tid=0' = 'object_number=0, tag_tid=41, entry_tid=0'
2024-05-08T01:21:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=41, entry_tid=0' '!=' ''
2024-05-08T01:21:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:21:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=41, entry_tid=0' = ''
2024-05-08T01:21:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=0, tag_tid=41, entry_tid=0'
2024-05-08T01:21:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=44, entry_tid=0'
2024-05-08T01:21:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:21:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:21:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=44
2024-05-08T01:21:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=41, entry_tid=0'
2024-05-08T01:21:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:21:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:21:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=41
2024-05-08T01:21:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=44, entry_tid=0'
2024-05-08T01:21:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:21:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:21:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=0
2024-05-08T01:21:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=41, entry_tid=0'
2024-05-08T01:21:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:21:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:21:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=0
2024-05-08T01:21:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 44 = 41 -a 0 -le 0
2024-05-08T01:21:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:21:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:21:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:21:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:21:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:21:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:21:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:21:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:21:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:21:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:21:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:21:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:21:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:21:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:21:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:21:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:21:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:21:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:21:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:21:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:21:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:21:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:21:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:21:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:21:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:21:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:21:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:21:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:21:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:21:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:21:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:21:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:21:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:21:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:21:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:21:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:21:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:08.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:08.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:21:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:21:08.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:08.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:21:08.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:21:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:21:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:21:08.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:21:08.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:21:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:21:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:21:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:21:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:21:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=44, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:21:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=44, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:21:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:08.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=44, entry_tid=0'
2024-05-08T01:21:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:21:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:21:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:21:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:21:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:21:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:21:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:21:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:21:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:21:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:21:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=44, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:21:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=44, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:21:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=44, entry_tid=0'
2024-05-08T01:21:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=44, entry_tid=0' -a 'object_number=0, tag_tid=44, entry_tid=0' = 'object_number=0, tag_tid=44, entry_tid=0'
2024-05-08T01:21:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:21:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:21:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:21:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:21:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:21:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:21:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:21:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:09
2024-05-08T01:21:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:21:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:21:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":41},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":44}}
2024-05-08T01:21:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:07'
2024-05-08T01:21:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:21:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:21:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:21:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:21:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:21:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:21:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:21:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:21:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:21:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:21:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:21:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:21:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:09'
2024-05-08T01:21:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:21:09'
2024-05-08T01:21:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:21:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:21:09'
2024-05-08T01:21:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:21:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:21:09</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:41},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:44}}</description><last_update>2024-05-08 01:21:07</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:21:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:21:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:21:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:09'
2024-05-08T01:21:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:21:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:21:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:21:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:21:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:21:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:21:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:21:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:21:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:21:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:21:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:21:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":41},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":44}}
2024-05-08T01:21:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:07
2024-05-08T01:21:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:21:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:21:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:21:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:09'
2024-05-08T01:21:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:21:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:21:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:21:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":41},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":44}}
2024-05-08T01:21:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:21:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:21:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:21:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:21:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:21:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:21:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:21:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:21:10.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:07'
2024-05-08T01:21:10.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:21:07'
2024-05-08T01:21:10.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:10.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:10.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:10.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:21:10.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:21:07'
2024-05-08T01:21:10.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:10.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:10.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:21:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:41},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:44}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:21:07</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:21:09</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:21:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:21:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:21:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:07'
2024-05-08T01:21:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:21:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:21:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:21:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:21:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:21:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:21:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:21:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:21:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:21:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:21:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:21:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:21:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:21:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:21:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:21:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:21:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:21:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:21:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:21:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:21:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:21:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:21:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:21:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:21:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:21:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:21:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:21:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:21:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:21:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:21:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:21:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:21:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:21:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:21:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:21:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:21:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:21:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:21:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:21:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:21:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:21:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:21:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:21:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:21:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:21:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:21:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:21:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:21:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:21:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:21:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:21:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:21:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:21:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:21:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:21:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:21:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:21:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:09
2024-05-08T01:21:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:21:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:21:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:21:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:12'
2024-05-08T01:21:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:21:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:21:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:21:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:21:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:21:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:21:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:21:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:14
2024-05-08T01:21:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:21:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:21:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:21:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:12'
2024-05-08T01:21:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:21:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:21:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:21:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:21:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:21:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:21:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:21:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:21:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:21:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:21:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:14'
2024-05-08T01:21:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:21:14'
2024-05-08T01:21:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:21:14'
2024-05-08T01:21:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:21:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:21:14</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:21:12</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:21:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:21:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:21:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:14'
2024-05-08T01:21:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:21:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:21:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:21:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:21:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:21:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:21:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:21:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:21:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:12
2024-05-08T01:21:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:21:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:21:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:21:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:14'
2024-05-08T01:21:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:21:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:21:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:21:16.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:21:16.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:21:16.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:16.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:21:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:21:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:21:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:21:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:21:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:12'
2024-05-08T01:21:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:21:12'
2024-05-08T01:21:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:21:12'
2024-05-08T01:21:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:21:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:21:12</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:21:14</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:21:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:21:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:21:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:12'
2024-05-08T01:21:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:21:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:21:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:21:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:21:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:21:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:21:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:21:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:21:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:21:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:21:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:21:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:21:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:21:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:21:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:21:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:21:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:21:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:21:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:21:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:21:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:21:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:21:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:21:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:21:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:21:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:21:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:21:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:21:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:21:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:21:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:21:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:21:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:21:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:21:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:21:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:21:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:21:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:21:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:21:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:21:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:21:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:21:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:21:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:21:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:21:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:21:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:21:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:21:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:21:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:21:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:21:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:21:19.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:21:19.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:19.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:19.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:21:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:21:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:21:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:21:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:21:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:21:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:21:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:21:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:21:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:21:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:21:19.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:19.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:21:19.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:21:19.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:21:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:21:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:21:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:21:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:21:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:21:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:21:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:21:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:21:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:21:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:21:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:21:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:21:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:21:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:21:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:21:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:21:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:21:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:21:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:21:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:21:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:21:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:21:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:21:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:21:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:21:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:21:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:21:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:21:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:21:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:21:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:21:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:21:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:21:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:21:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:21:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:21:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:21:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:21:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:21:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:21:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:21:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:21:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:21:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:21:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:21:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:21:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:21:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:21:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:21:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:21:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:21:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:21:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:21:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:21:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:21:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:21:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:21:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:21:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:21:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:21:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:21:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:21:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:21:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:21:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:21:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:21:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:21:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:21:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:21:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:21:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:21:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:21:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:21:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:21:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:21:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:21:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:21:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:21:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:21:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:21:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:21:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:21:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:21:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=45, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:21:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=48, entry_tid=0'
2024-05-08T01:21:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:21:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:21:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:21:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:21:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:21:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:21:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:21:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:21:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:21:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:21:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:21:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=45, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:21:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=45, entry_tid=0'
2024-05-08T01:21:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=48, entry_tid=0' -a 'object_number=0, tag_tid=48, entry_tid=0' = 'object_number=0, tag_tid=45, entry_tid=0'
2024-05-08T01:21:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=45, entry_tid=0' '!=' ''
2024-05-08T01:21:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:21:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=45, entry_tid=0' = ''
2024-05-08T01:21:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=0, tag_tid=45, entry_tid=0'
2024-05-08T01:21:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=48, entry_tid=0'
2024-05-08T01:21:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:21:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:21:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=48
2024-05-08T01:21:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=45, entry_tid=0'
2024-05-08T01:21:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:21:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:21:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=45
2024-05-08T01:21:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=48, entry_tid=0'
2024-05-08T01:21:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:21:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:21:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=0
2024-05-08T01:21:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=45, entry_tid=0'
2024-05-08T01:21:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:21:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:21:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=0
2024-05-08T01:21:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 48 = 45 -a 0 -le 0
2024-05-08T01:21:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:21:20.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:21:20.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:21:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:21:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:21:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:21:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:21:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:21:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:21:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:21:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:21:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:21:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:21:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:21:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:21:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:21:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:21:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:21:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:21:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:21:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:21:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:21:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:21:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:21:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:21:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:21:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:21:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:21:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:21:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:21:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:21:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:21:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:21:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:21:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:21:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:21:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:21:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:21:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:21:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:21:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:21:20.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:21:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:21:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:21:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:21:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:21:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=48, entry_tid=0'
2024-05-08T01:21:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:21:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:21:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:21:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:21:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:21:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:21:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:21:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:21:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:21:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:21:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:21:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:21:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=48, entry_tid=0'
2024-05-08T01:21:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=48, entry_tid=0' -a 'object_number=0, tag_tid=48, entry_tid=0' = 'object_number=0, tag_tid=48, entry_tid=0'
2024-05-08T01:21:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:21:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:21:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:21:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:21:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:21:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:21:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:21:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:17
2024-05-08T01:21:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:21:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:21:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":45},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":48}}
2024-05-08T01:21:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:19'
2024-05-08T01:21:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:21:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:21:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:21:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:21:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:21:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:21:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:21:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:21:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:21:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:21:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:21:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:21:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:17'
2024-05-08T01:21:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:21:17'
2024-05-08T01:21:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:21:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:21:17'
2024-05-08T01:21:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:21:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:21:17</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:45},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:48}}</description><last_update>2024-05-08 01:21:19</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:21:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:21:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:21:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:17'
2024-05-08T01:21:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:21:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:21:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:21:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:21:21.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:21:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:21:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:21:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:21:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:21:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:21:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:21:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":45},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":48}}
2024-05-08T01:21:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:19
2024-05-08T01:21:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:21:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:21:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:21:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:22'
2024-05-08T01:21:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:21:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:21:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:21:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":45},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":48}}
2024-05-08T01:21:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:21:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:21:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:21:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:21:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:21:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:21:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:21:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:21:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:19'
2024-05-08T01:21:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:21:19'
2024-05-08T01:21:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:21:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:21:19'
2024-05-08T01:21:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:21:23.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:45},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:48}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:21:19</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:21:22</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:21:23.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:21:23.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:21:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:19'
2024-05-08T01:21:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:21:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:21:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:21:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:21:23.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:21:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:21:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:21:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:21:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:21:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:21:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:21:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:21:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:21:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:21:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:21:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:21:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:21:24.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:21:24.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:21:24.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:24.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:24.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:21:24.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:21:24.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:24.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:21:24.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:21:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:21:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:21:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:21:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:21:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:21:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:21:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:21:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:21:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:21:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:21:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:21:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:21:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:21:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:21:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:21:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:21:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:21:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:21:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:21:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:21:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:21:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:21:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:21:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:21:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:21:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:21:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:21:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:21:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:21:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:21:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:21:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:21:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:21:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:21:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:21:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:21:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:21:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:21:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:21:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:21:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:21:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:21:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:21:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:21:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:21:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:21:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:21:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:21:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:21:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:21:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:21:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:21:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:21:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:21:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:21:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:21:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:21:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:21:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:21:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:21:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:21:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:21:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:21:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:21:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:21:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:21:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:21:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:21:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:21:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:21:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:21:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:21:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:21:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:21:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:21:27.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:21:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:21:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:25
2024-05-08T01:21:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:21:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:21:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:21:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:27'
2024-05-08T01:21:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:21:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:21:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:21:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:21:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:21:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:21:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:21:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:29
2024-05-08T01:21:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:21:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:21:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:21:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:27'
2024-05-08T01:21:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:21:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:21:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:21:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:21:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:21:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:21:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:21:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:21:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:21:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:21:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:29'
2024-05-08T01:21:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:21:29'
2024-05-08T01:21:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:21:29'
2024-05-08T01:21:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:21:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:21:29</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:21:27</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:21:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:21:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:21:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:29'
2024-05-08T01:21:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:21:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:21:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:21:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:21:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:21:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:21:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:21:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:27
2024-05-08T01:21:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:21:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:21:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:21:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:29'
2024-05-08T01:21:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:21:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:21:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:21:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:21:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:21:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:21:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:21:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:21:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:21:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:21:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:27'
2024-05-08T01:21:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:21:27'
2024-05-08T01:21:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:21:27'
2024-05-08T01:21:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:21:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:21:27</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:21:29</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:21:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:21:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:21:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:27'
2024-05-08T01:21:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:21:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:21:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:21:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:21:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:21:30.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:21:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:21:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:21:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:21:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:21:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:21:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:21:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:21:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:21:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:21:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:21:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:21:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:21:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:21:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:21:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:21:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:21:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:21:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:21:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:21:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:21:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:21:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:21:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:21:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:21:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:21:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:21:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:21:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:21:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:21:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:21:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:21:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:21:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:21:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:21:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:21:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:21:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:21:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:21:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:21:31.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:31.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:21:32.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:21:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Starting"'
2024-05-08T01:21:32.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Starting"'
2024-05-08T01:21:32.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:21:32.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:21:32.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:32.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:21:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:21:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:21:34.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:21:34.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:21:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:21:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:21:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:21:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:21:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:21:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:21:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:21:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:21:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:21:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:21:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:21:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:21:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:21:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:21:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:21:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:21:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:21:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:21:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:21:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:21:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:21:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:21:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:21:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:21:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:21:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:21:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:21:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:21:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:21:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:21:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:21:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:21:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:21:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:21:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:21:34.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:21:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:21:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:21:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:21:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:21:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:21:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:21:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:21:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:21:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:21:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:21:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:21:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:21:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:21:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:21:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:21:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:21:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:21:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:21:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:21:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:21:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:21:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:21:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:21:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:21:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:21:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:21:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:21:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:21:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:21:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:21:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:21:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:21:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:21:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:21:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:21:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:21:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:21:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:21:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:21:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:21:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:21:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:21:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:21:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:21:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:21:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:21:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:21:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:21:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:21:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:21:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:21:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:21:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:21:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:21:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:21:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:21:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:21:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=49, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:21:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=49, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:21:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=49, entry_tid=0'
2024-05-08T01:21:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:21:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:21:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:21:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:21:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:21:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:21:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:21:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:34.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:21:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:21:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:21:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=49, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:21:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=49, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:21:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=49, entry_tid=0'
2024-05-08T01:21:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=49, entry_tid=0' -a 'object_number=0, tag_tid=49, entry_tid=0' = 'object_number=0, tag_tid=49, entry_tid=0'
2024-05-08T01:21:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:21:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:21:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:21:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:21:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:21:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:21:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:21:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:34
2024-05-08T01:21:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:21:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:21:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":46},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":49}}
2024-05-08T01:21:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:32'
2024-05-08T01:21:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:21:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:21:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:21:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:21:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:21:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:21:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:21:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:21:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:21:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:21:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:21:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:21:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:34'
2024-05-08T01:21:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:21:34'
2024-05-08T01:21:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:21:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:21:34'
2024-05-08T01:21:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:21:35.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:21:34</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:46},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:49}}</description><last_update>2024-05-08 01:21:32</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:21:35.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:21:35.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:21:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:34'
2024-05-08T01:21:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:21:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:21:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:21:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:21:35.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:21:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:21:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:21:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:21:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:21:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:21:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:21:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":46},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":49}}
2024-05-08T01:21:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:32
2024-05-08T01:21:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:21:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:21:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:21:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:34'
2024-05-08T01:21:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:21:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:21:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:21:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":46},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":49}}
2024-05-08T01:21:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:21:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:21:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:21:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:21:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:21:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:21:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:21:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:21:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:32'
2024-05-08T01:21:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:21:32'
2024-05-08T01:21:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:21:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:21:32'
2024-05-08T01:21:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:21:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:12.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.5,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:49},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:49}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:21:37</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:21:34</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:21:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:21:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:21:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:37'
2024-05-08T01:21:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:21:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:21:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:21:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:21:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:21:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:21:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:21:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:21:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:21:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:21:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:21:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:21:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:21:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:21:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:21:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:21:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:21:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:21:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:21:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:21:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:21:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:21:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:21:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:21:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:21:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:21:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:21:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:21:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:21:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:21:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:21:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:21:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:21:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:21:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:21:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:21:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:21:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:21:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:21:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:21:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:21:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:21:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:21:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:21:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:21:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:21:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:21:38.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:21:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:21:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:21:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:21:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:21:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:21:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:21:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:21:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:21:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:21:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:21:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:21:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:21:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:21:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:21:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:21:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:21:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:21:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:21:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:21:40.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:40.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:21:40.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:21:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:21:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:21:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:21:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:21:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:21:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:21:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:21:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:21:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:21:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:21:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:21:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:21:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:21:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:21:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:21:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:21:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:21:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:21:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:21:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:21:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:21:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:21:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:21:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:21:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:21:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:21:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:21:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:21:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:21:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:21:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:21:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:39
2024-05-08T01:21:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:21:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:21:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:21:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:38'
2024-05-08T01:21:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:21:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:21:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:21:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:21:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:21:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:21:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:21:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:21:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:21:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:21:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:39'
2024-05-08T01:21:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:21:39'
2024-05-08T01:21:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:21:39'
2024-05-08T01:21:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:21:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:21:39</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>stopped</description><last_update>2024-05-08 01:21:38</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:21:41.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:21:41.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:21:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:39'
2024-05-08T01:21:41.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:21:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:21:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:21:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:21:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:21:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:21:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:21:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:21:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:42
2024-05-08T01:21:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:21:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:21:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:21:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:39'
2024-05-08T01:21:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:21:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:21:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:21:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:21:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:21:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:21:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:21:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:21:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:21:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:21:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:42'
2024-05-08T01:21:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:21:42'
2024-05-08T01:21:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:21:42'
2024-05-08T01:21:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:21:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:21:42</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:21:39</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:21:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:21:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:21:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:42'
2024-05-08T01:21:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:21:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:21:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:21:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:43.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:43.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:43.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:21:43.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:43.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:43.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:43.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:21:43.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:21:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:21:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:21:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:21:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:21:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:21:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:21:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:21:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:21:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:21:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:21:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:21:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:21:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:21:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:21:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:21:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:21:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:21:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:21:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:21:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:21:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:21:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:21:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:21:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:21:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:21:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:21:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:21:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:21:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:21:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:21:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:21:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:21:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:21:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:21:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:21:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:21:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:21:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:21:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:21:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:21:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:21:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Starting"'
2024-05-08T01:21:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Starting"'
2024-05-08T01:21:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:21:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:21:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:21:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:21:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:21:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:21:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:21:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:21:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:21:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:21:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:21:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:21:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:21:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:21:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:21:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:21:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:21:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:21:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:21:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:21:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:21:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:21:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:21:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:21:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:21:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:21:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:21:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:21:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:21:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:21:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:21:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:21:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:21:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:21:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:21:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:21:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:21:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:21:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:21:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:21:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:21:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:21:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:21:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:21:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:21:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:21:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:21:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:21:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:21:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:21:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:21:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:21:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:21:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:21:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:21:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:21:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:21:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:21:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:21:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:21:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:21:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:21:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:21:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:21:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:21:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:21:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:21:46.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:46.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:21:46.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:21:46.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:21:46.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:21:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:21:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:21:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:21:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:21:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:21:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:21:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:21:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:21:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:21:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:46.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:21:46.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:21:46.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:46.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:46.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:21:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:21:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:21:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:21:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:21:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:21:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:21:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:21:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:21:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:21:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:21:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:21:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:21:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:21:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:21:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:21:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:21:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:21:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=53, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:21:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=53, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:21:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=53, entry_tid=0'
2024-05-08T01:21:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:21:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:21:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:21:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:21:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:21:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:21:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:21:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:21:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:21:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:21:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:21:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:21:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=53, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:21:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=53, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:21:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:21:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=53, entry_tid=0'
2024-05-08T01:21:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=53, entry_tid=0' -a 'object_number=0, tag_tid=53, entry_tid=0' = 'object_number=0, tag_tid=53, entry_tid=0'
2024-05-08T01:21:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:21:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:21:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:21:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:21:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:47.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:47.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:21:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:21:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:21:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:47
2024-05-08T01:21:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:21:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:21:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":50},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":53}}
2024-05-08T01:21:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:44'
2024-05-08T01:21:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:21:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:21:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:21:47.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:21:47.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:21:47.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:21:47.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:21:48.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:21:48.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:21:48.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:21:48.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:21:48.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:21:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:47'
2024-05-08T01:21:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:21:47'
2024-05-08T01:21:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:21:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:21:47'
2024-05-08T01:21:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:48.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:21:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:21:47</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:50},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:53}}</description><last_update>2024-05-08 01:21:44</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:21:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:21:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:21:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:47'
2024-05-08T01:21:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:21:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:21:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:21:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:21:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:21:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:21:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:21:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:21:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:21:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:21:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:21:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":50},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":53}}
2024-05-08T01:21:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:44
2024-05-08T01:21:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:21:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:21:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:21:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:47'
2024-05-08T01:21:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:21:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:21:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:21:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":50},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":53}}
2024-05-08T01:21:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:21:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:21:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:21:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:21:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:21:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:21:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:21:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:21:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:44'
2024-05-08T01:21:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:21:44'
2024-05-08T01:21:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:21:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:21:44'
2024-05-08T01:21:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:21:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:50},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:53}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:21:44</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:21:47</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:21:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:21:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:21:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:44'
2024-05-08T01:21:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:21:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:21:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:21:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:21:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:21:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:21:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:21:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:21:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:21:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:21:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:21:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:21:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:21:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:21:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:21:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:21:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:21:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:21:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:21:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:21:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:21:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:21:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:21:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:21:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:21:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:21:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:21:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:21:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:21:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:21:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:21:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:50.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:50.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:21:50.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:21:50.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:50.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:50.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:21:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:21:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:21:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:21:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:21:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:21:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:21:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:21:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:21:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:21:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:21:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:21:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:21:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:21:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:21:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:21:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:21:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:21:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:21:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:21:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:21:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:21:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:21:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:21:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:21:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:21:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:21:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:21:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:21:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:21:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:21:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:21:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:21:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:21:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:21:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:21:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:21:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:21:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:21:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:21:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:21:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:21:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:21:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:21:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:21:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:21:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:21:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:21:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:21:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:21:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:21:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:21:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:21:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:21:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:21:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:21:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:21:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:21:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:21:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:21:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:21:52.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:21:52.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:21:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:21:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:21:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:21:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:21:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:53.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:21:53.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:53.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:53.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:53.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:53.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:53.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:53.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:53.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:53.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:21:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:21:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:21:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:51
2024-05-08T01:21:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:21:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:21:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:21:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:52'
2024-05-08T01:21:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:21:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:21:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:21:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:21:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:21:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:21:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:21:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:54
2024-05-08T01:21:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:21:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:21:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:21:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:52'
2024-05-08T01:21:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:21:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:21:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:21:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:21:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:21:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:21:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:21:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:21:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:21:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:21:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:54'
2024-05-08T01:21:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:21:54'
2024-05-08T01:21:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:21:54'
2024-05-08T01:21:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:21:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:21:54</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:21:52</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:21:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:21:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:21:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:54'
2024-05-08T01:21:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:21:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:21:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:21:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:21:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:21:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:21:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:21:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:21:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:21:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:21:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:21:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:21:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:21:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:21:52
2024-05-08T01:21:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:21:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:21:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:21:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:21:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:21:54'
2024-05-08T01:21:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:21:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:21:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:21:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:21:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:21:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:21:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:57.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:21:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:21:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:21:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:21:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:52'
2024-05-08T01:21:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:21:52'
2024-05-08T01:21:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:21:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:21:52'
2024-05-08T01:21:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:21:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:21:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:21:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:21:52</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:21:54</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:21:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:21:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:21:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:21:52'
2024-05-08T01:21:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:21:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:21:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:21:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:21:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:21:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:21:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:21:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:21:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:21:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:21:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:21:57.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:21:57.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:57.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:21:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:21:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:21:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:21:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:21:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:21:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:21:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:21:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:21:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:21:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:21:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:21:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:21:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:21:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:21:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:21:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:21:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:21:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:21:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:21:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:21:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:21:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:21:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:21:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:21:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:21:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:21:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:21:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:21:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:21:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:21:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:21:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:21:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:21:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:21:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:21:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:21:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:21:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:21:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:21:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:21:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:21:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:21:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:21:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:21:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:21:58.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:21:58.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:21:58.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:21:58.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:21:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:21:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:21:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:21:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:21:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:21:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:21:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:21:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:21:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:22:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:22:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:22:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:22:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:22:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:22:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:22:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:22:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:22:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:22:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:22:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:22:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:22:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:22:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:22:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:22:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:22:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:22:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:22:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:22:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:22:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:22:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:22:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:22:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:22:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:22:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:22:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:22:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:22:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:22:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:22:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:22:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:22:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:22:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:22:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:22:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:22:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:22:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:22:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 4
2024-05-08T01:22:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:22:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:22:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:22:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:22:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:22:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:22:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:22:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:22:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:22:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:22:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:22:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:22:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:22:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:22:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:22:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:22:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:22:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:22:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:22:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:22:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:22:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:22:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:22:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:22:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:22:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:22:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:22:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:22:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:22:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:22:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:22:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:22:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:22:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:22:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:22:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:22:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:22:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:22:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:22:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:22:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:22:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:22:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:22:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:22:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:22:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:22:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:22:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:22:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:22:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:22:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:22:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:22:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:22:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:22:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:22:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:22:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:22:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:22:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:22:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:22:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:22:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:22:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:22:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:22:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:22:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:22:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:22:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:22:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:22:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:22:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:22:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:22:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:22:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:22:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:22:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:22:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:22:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:22:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:22:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:22:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:22:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:22:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:22:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:22:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:22:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:22:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:22:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:22:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:22:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:22:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:22:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:22:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:22:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:22:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:22:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=54, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:22:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=54, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:22:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=54, entry_tid=0'
2024-05-08T01:22:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:22:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:22:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:22:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:22:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:22:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:22:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:22:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:22:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:22:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:22:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=54, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:22:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=54, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:22:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=54, entry_tid=0'
2024-05-08T01:22:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=54, entry_tid=0' -a 'object_number=0, tag_tid=54, entry_tid=0' = 'object_number=0, tag_tid=54, entry_tid=0'
2024-05-08T01:22:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:22:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:22:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:22:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:22:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:22:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:22:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:22:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:04
2024-05-08T01:22:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:22:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:22:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":51},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54}}
2024-05-08T01:22:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:02'
2024-05-08T01:22:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:22:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:22:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:22:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:22:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:22:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:22:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:22:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:22:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:22:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:22:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:22:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:22:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:04'
2024-05-08T01:22:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:22:04'
2024-05-08T01:22:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:22:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:22:04'
2024-05-08T01:22:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:22:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:22:04</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:51},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:54}}</description><last_update>2024-05-08 01:22:02</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:22:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:22:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:22:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:04'
2024-05-08T01:22:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:22:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:22:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:22:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:22:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:22:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:22:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:22:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:06.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:06.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:22:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:22:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:22:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:22:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":8.0,"entries_behind_primary":0,"entries_per_second":0.33,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54}}
2024-05-08T01:22:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:07
2024-05-08T01:22:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:22:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:22:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:22:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:04'
2024-05-08T01:22:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:22:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:22:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:22:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":8.0,"entries_behind_primary":0,"entries_per_second":0.33,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54}}
2024-05-08T01:22:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:22:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:22:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:22:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:22:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:22:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:22:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:22:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:22:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:07'
2024-05-08T01:22:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:22:07'
2024-05-08T01:22:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:22:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:22:07'
2024-05-08T01:22:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:22:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:8.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.33,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:54},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:54}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:22:07</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:22:04</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:22:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:22:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:22:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:07'
2024-05-08T01:22:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:22:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:22:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:22:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:22:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:22:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:22:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:22:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:22:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:22:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:22:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:22:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:22:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:22:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:22:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:22:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:22:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:22:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:22:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:22:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:22:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:22:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:22:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:22:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:22:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:22:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:22:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:22:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:22:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:22:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:22:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:22:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:22:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:22:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:22:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:22:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:22:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:22:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:22:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:22:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:22:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:22:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:22:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:22:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:22:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:22:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:22:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:22:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:22:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:22:08.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:22:08.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:22:08.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:22:08.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:08.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:22:10.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:22:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:22:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:22:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:22:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:22:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:22:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:22:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:22:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:22:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:22:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:22:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:22:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:22:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:22:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:22:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:22:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:22:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:22:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:22:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:22:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:22:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:22:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:22:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:22:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:22:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:22:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:22:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:22:10.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:22:10.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:22:10.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:22:10.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:22:10.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:10.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:10.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:10.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:22:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:22:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:22:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:22:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:22:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:22:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:22:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:22:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:22:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:22:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:22:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:09
2024-05-08T01:22:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:22:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:22:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:22:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:08'
2024-05-08T01:22:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:22:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:22:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:22:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:22:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:22:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:22:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:22:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:22:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:22:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:22:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:22:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:22:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:09'
2024-05-08T01:22:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:22:09'
2024-05-08T01:22:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:22:09'
2024-05-08T01:22:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:22:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:22:09</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:22:12</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:22:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:22:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:22:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:09'
2024-05-08T01:22:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:22:12.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:22:12.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:22:12.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:22:12.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:22:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:22:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:22:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:22:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:22:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:22:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:12
2024-05-08T01:22:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:22:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:22:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:22:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:09'
2024-05-08T01:22:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:22:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:22:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:22:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:22:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:22:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:22:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:22:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:22:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:22:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:22:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:22:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:22:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:12'
2024-05-08T01:22:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:22:12'
2024-05-08T01:22:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:22:12'
2024-05-08T01:22:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:22:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:22:12</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:22:09</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:22:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:22:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:22:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:12'
2024-05-08T01:22:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:22:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:22:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:22:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:22:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:22:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:22:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:22:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:22:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:22:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:22:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:22:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:22:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:22:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:22:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:22:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:22:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:22:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:22:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:22:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:22:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:22:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:22:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:22:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:22:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:22:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:22:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:22:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:22:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:22:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:22:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:22:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:22:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:22:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:22:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:22:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:22:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:22:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:22:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:22:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:22:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:22:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:22:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:22:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:22:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:22:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:22:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:22:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:22:14.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:22:14.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:22:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:22:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:22:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:22:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:22:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:22:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:22:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:22:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:22:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:22:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:22:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:22:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:22:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:22:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:22:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:22:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:22:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:22:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:22:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:22:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:22:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:22:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:22:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:22:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:22:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:22:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:22:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:22:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:22:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:22:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:22:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:22:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:22:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:22:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:22:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:22:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:22:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:22:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:22:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:22:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:22:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:22:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:22:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:22:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:22:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:22:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:22:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:22:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:22:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:22:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:22:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:22:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:22:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:22:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:22:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:22:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:22:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:22:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:22:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:22:14.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:22:14.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:22:14.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=55, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:22:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=58, entry_tid=0'
2024-05-08T01:22:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:22:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:22:14.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:14.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:14.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:14.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:22:14.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:22:14.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:22:14.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:22:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:22:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:22:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:22:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:22:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:22:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=55, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:22:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=55, entry_tid=0'
2024-05-08T01:22:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=58, entry_tid=0' -a 'object_number=0, tag_tid=58, entry_tid=0' = 'object_number=0, tag_tid=55, entry_tid=0'
2024-05-08T01:22:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=55, entry_tid=0' '!=' ''
2024-05-08T01:22:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:22:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=55, entry_tid=0' = ''
2024-05-08T01:22:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=0, tag_tid=55, entry_tid=0'
2024-05-08T01:22:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=58, entry_tid=0'
2024-05-08T01:22:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:22:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:22:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=58
2024-05-08T01:22:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=55, entry_tid=0'
2024-05-08T01:22:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:22:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:22:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=55
2024-05-08T01:22:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=58, entry_tid=0'
2024-05-08T01:22:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:22:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:22:14.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=0
2024-05-08T01:22:14.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=55, entry_tid=0'
2024-05-08T01:22:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:22:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:22:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=0
2024-05-08T01:22:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 58 = 55 -a 0 -le 0
2024-05-08T01:22:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:22:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:22:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:22:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:22:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:22:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:22:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:22:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:22:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:22:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:22:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:22:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:22:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:22:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:22:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:22:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:22:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:22:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:22:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:22:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:22:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:22:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:22:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:22:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:22:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:22:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:22:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:22:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:22:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:22:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:22:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:22:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:22:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:22:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:22:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:22:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:22:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:22:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:22:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:22:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:22:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:22:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:22:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:22:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:22:15.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:15.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:22:15.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:22:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:22:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:22:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:22:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:22:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=58, entry_tid=0'
2024-05-08T01:22:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:22:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:22:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:22:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:22:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:22:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:22:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:22:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:22:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:22:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:22:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:22:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:22:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=58, entry_tid=0'
2024-05-08T01:22:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=58, entry_tid=0' -a 'object_number=0, tag_tid=58, entry_tid=0' = 'object_number=0, tag_tid=58, entry_tid=0'
2024-05-08T01:22:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:22:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:22:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:22:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:22:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:22:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:22:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:22:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:12
2024-05-08T01:22:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:22:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:22:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":55},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":58}}
2024-05-08T01:22:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:14'
2024-05-08T01:22:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:22:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:22:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:22:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:22:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:22:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:22:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:22:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:22:18.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:18.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:17
2024-05-08T01:22:18.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:18.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:22:18.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:22:18.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":55},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":58}}
2024-05-08T01:22:18.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:14'
2024-05-08T01:22:18.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:22:18.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:22:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:22:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:22:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:22:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:22:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:22:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:22:18.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:22:18.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:22:18.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:22:18.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:22:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:17'
2024-05-08T01:22:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:22:17'
2024-05-08T01:22:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:22:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:22:17'
2024-05-08T01:22:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:22:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:22:17</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:55},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:58}}</description><last_update>2024-05-08 01:22:14</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:22:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:22:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:22:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:17'
2024-05-08T01:22:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:22:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:22:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:22:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:22:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:22:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:22:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:22:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:22:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:22:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:22:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:22:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":55},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":58}}
2024-05-08T01:22:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:14
2024-05-08T01:22:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:22:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:22:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:22:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:17'
2024-05-08T01:22:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:22:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:22:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:22:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":55},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":58}}
2024-05-08T01:22:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:22:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:22:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:22:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:22:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:22:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:22:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:22:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:22:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:14'
2024-05-08T01:22:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:22:14'
2024-05-08T01:22:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:22:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:22:14'
2024-05-08T01:22:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:19.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:22:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:55},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:58}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:22:14</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:22:17</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:22:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:22:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:22:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:14'
2024-05-08T01:22:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:22:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:22:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:22:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:22:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:22:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:22:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:22:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:22:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:22:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:22:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:22:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:22:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:22:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:22:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:22:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:22:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:22:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:22:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:22:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:22:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:22:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:22:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:22:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:22:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:22:20.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:20.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:22:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:22:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:22:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:22:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:22:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:22:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:22:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:22:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:22:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:22:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:22:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:22:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:22:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:22:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:22:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:22:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:22:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:22:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:22:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:22:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:22:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:22:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:22:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:22:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:22:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:22:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:22:21.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:22:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:22:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:22:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:22:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:22:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:22:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:22:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:22:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:22:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:22:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:22:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:22:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:22:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:22:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:22:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:22:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:22:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:22:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:22:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:22:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:22:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:22:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:22:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:22:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:22:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:22:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:22:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:22:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:22:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:22:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:22:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:22:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:22:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:22:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:22:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:22:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:22:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:22:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:22:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:22:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:22:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:22:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:22:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:22:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:22:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:22:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:24.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:24.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:24.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:24.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:24.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:22:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:22:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:22:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:24
2024-05-08T01:22:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:22:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:22:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:22:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:22'
2024-05-08T01:22:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:22:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:22:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:22:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:22:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:22:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:22:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:22:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:22:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:22:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:22:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:22:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:22:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:24'
2024-05-08T01:22:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:22:24'
2024-05-08T01:22:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:22:24'
2024-05-08T01:22:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:22:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:22:24</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:22:22</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:22:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:22:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:22:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:24'
2024-05-08T01:22:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:22:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:22:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:22:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:22:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:22:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:22:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:22:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:22:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:22:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:22:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:22
2024-05-08T01:22:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:22:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:22:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:22:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:24'
2024-05-08T01:22:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:22:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:22:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:22:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:22:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:22:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:22:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:22:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:22:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:22:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:22:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:22:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:22:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:22'
2024-05-08T01:22:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:22:22'
2024-05-08T01:22:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:22:22'
2024-05-08T01:22:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:22:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:22:22</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:22:24</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:22:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:22:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:22:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:22'
2024-05-08T01:22:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:22:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:22:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:22:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:22:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:22:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:22:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:22:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:22:25.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:22:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:22:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:22:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:22:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:22:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:22:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:22:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:22:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:22:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:22:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:22:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:22:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:22:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:22:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:22:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:22:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:22:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:22:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:22:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:22:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:22:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:22:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:22:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:22:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:22:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:22:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:22:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:22:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:22:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:22:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:22:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:22:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:22:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:22:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:22:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:22:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:22:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:22:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:22:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:22:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:22:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:22:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:22:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:22:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:22:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:22:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:22:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:22:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:22:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:22:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:22:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:22:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:22:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:22:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:22:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:22:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:22:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:22:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:22:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:22:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:22:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:22:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:22:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:22:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:22:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:22:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:22:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:22:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:22:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:22:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:22:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:22:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:22:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:22:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:22:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:22:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:22:28.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:22:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:22:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:22:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:22:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:22:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:22:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:22:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:22:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:22:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:22:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:22:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:22:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:22:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:22:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:22:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:22:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:22:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:22:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:22:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:22:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:22:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:22:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:22:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:22:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:22:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:22:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:22:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:22:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:22:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:22:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:22:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:22:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:22:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:22:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:22:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:22:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:22:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:22:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:22:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:22:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:22:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:22:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:22:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:22:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:22:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:22:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:22:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:22:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:22:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:22:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:22:29.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:29.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:29.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:29.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:22:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:22:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:22:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:22:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:22:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:22:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:22:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:22:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:22:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:22:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:22:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=59, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:22:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=59, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:22:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:29.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=59, entry_tid=0'
2024-05-08T01:22:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:22:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:22:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:22:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:22:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:22:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:22:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:22:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:22:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:22:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:22:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=59, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:22:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=59, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:22:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=59, entry_tid=0'
2024-05-08T01:22:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=59, entry_tid=0' -a 'object_number=0, tag_tid=59, entry_tid=0' = 'object_number=0, tag_tid=59, entry_tid=0'
2024-05-08T01:22:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:22:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:22:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:22:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:22:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:22:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:29
2024-05-08T01:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":56},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":59}}
2024-05-08T01:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:27'
2024-05-08T01:22:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:22:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:22:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:22:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:22:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:22:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:22:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:22:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:22:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:22:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:22:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:22:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:22:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:29'
2024-05-08T01:22:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:22:29'
2024-05-08T01:22:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:22:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:22:29'
2024-05-08T01:22:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:22:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:22:29</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:56},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:59}}</description><last_update>2024-05-08 01:22:27</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:22:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:22:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:22:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:29'
2024-05-08T01:22:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:22:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:22:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:22:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:22:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:22:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:22:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:22:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:22:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:22:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:22:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:22:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":56},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":59}}
2024-05-08T01:22:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:27
2024-05-08T01:22:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:22:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:22:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:22:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:29'
2024-05-08T01:22:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:22:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:22:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:22:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":56},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":59}}
2024-05-08T01:22:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:22:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:22:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:22:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:22:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:22:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:22:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:22:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:22:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:27'
2024-05-08T01:22:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:22:27'
2024-05-08T01:22:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:22:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:22:27'
2024-05-08T01:22:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:22:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:56},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:59}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:22:27</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:22:29</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:22:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:22:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:22:31.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:27'
2024-05-08T01:22:31.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:22:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:22:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:22:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:22:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:22:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:22:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:22:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:22:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:22:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:22:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:22:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:22:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:22:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:22:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:22:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:22:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:22:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:22:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:22:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:22:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:22:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:22:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:22:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:22:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:22:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:22:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:22:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:22:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:22:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:22:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:22:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:22:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:22:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:22:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:22:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:22:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:22:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:22:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:22:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:22:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:22:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:22:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:22:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:22:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:22:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:22:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:22:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:22:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:22:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:22:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:22:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:22:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:22:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:22:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:22:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:22:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:22:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:22:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:22:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:22:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:22:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:22:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:22:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:22:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:22:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:22:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:22:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:22:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:22:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:22:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:22:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:22:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:22:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:22:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:22:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:22:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:22:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:22:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:22:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:22:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:22:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:22:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:22:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:22:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:22:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:22:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:22:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:22:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:22:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:22:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:22:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:22:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:22:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:22:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:22:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:22:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:34
2024-05-08T01:22:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:22:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:22:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:22:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:33'
2024-05-08T01:22:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:22:36.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:22:36.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:22:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:22:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:22:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:22:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:22:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:22:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:22:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:22:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:22:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:22:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:34'
2024-05-08T01:22:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:22:34'
2024-05-08T01:22:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:22:34'
2024-05-08T01:22:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:22:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:22:34</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>stopped</description><last_update>2024-05-08 01:22:33</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:22:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:22:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:22:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:34'
2024-05-08T01:22:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:22:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:22:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:22:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:22:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:22:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:22:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:22:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:22:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:22:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:22:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:37
2024-05-08T01:22:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:22:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:22:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:22:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:34'
2024-05-08T01:22:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:22:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:22:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:22:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:22:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:22:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:22:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:22:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:22:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:22:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:22:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:22:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:22:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:37'
2024-05-08T01:22:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:22:37'
2024-05-08T01:22:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:22:37'
2024-05-08T01:22:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:22:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:22:37</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:22:34</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:22:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:22:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:22:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:37'
2024-05-08T01:22:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:22:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:22:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:22:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:22:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:22:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:22:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:22:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:22:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:22:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:22:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:22:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:22:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:22:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:22:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:22:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:22:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:22:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:22:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:22:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:22:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:22:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:22:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:22:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:22:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:22:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:22:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:22:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:22:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:22:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:22:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:22:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:22:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:22:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:22:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:22:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:22:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:22:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:22:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:22:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:22:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:22:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:22:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:22:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:22:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:22:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:22:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:22:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:38.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:38.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:22:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:22:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:22:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:22:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:22:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:22:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:22:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:22:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:22:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:22:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:22:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:22:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:22:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:22:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:22:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:22:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:22:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:22:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:22:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:22:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:22:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:22:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:22:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:22:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:22:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:22:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:22:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:22:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:22:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:22:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:22:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:22:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:22:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:22:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:22:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:22:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:22:40.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:22:40.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:22:40.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:22:40.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:22:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:22:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:22:40.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:22:40.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:22:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:22:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:22:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:22:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:22:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:22:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:22:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:22:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:22:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:22:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:22:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:22:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:22:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:22:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:22:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:22:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:22:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:22:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:22:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:22:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:22:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:22:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:22:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:22:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:22:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:22:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:22:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:22:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:22:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:22:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:22:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:22:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:22:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:22:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:22:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:22:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:22:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:22:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:22:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:22:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:22:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:22:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:22:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:22:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:22:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:22:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:22:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:22:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:22:41.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:41.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:41.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:41.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:22:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:22:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:22:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:22:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:22:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:22:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:22:41.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:41.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:22:41.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:22:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:22:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:22:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=63, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:22:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=63, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:22:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=63, entry_tid=0'
2024-05-08T01:22:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:22:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:22:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:22:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:22:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:22:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:22:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:22:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:22:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:22:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:22:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=63, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:22:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=63, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:22:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=63, entry_tid=0'
2024-05-08T01:22:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=63, entry_tid=0' -a 'object_number=0, tag_tid=63, entry_tid=0' = 'object_number=0, tag_tid=63, entry_tid=0'
2024-05-08T01:22:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:22:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:22:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:22:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:22:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:22:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:22:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:22:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:42
2024-05-08T01:22:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:22:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:22:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":60},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":63}}
2024-05-08T01:22:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:39'
2024-05-08T01:22:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:22:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:22:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:22:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:22:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:22:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:22:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:22:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:22:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:22:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:22:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:22:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:22:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:42'
2024-05-08T01:22:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:22:42'
2024-05-08T01:22:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:22:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:22:42'
2024-05-08T01:22:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:22:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:22:42</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:60},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:63}}</description><last_update>2024-05-08 01:22:39</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:22:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:22:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:22:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:42'
2024-05-08T01:22:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:22:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:22:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:22:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:22:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:22:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:22:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:22:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:22:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:22:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:22:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:22:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":60},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":63}}
2024-05-08T01:22:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:39
2024-05-08T01:22:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:22:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:22:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:22:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:42'
2024-05-08T01:22:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:22:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:22:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:22:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":60},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":63}}
2024-05-08T01:22:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:22:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:22:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:22:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:22:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:22:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:22:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:22:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:22:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:39'
2024-05-08T01:22:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:22:39'
2024-05-08T01:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:22:39'
2024-05-08T01:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:22:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:60},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:63}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:22:39</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:22:42</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:22:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:22:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:22:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:39'
2024-05-08T01:22:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:22:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:22:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:22:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:22:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:22:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:22:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:22:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:22:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:22:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:22:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:22:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:22:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:22:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:22:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:22:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:22:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:22:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:22:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:22:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:22:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:22:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:22:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:22:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:22:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:22:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:22:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:22:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:22:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:22:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:22:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:22:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:22:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:22:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:22:44.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:22:44.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:22:44.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:22:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:22:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:22:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:22:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:22:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:22:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:22:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:22:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:22:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:22:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:22:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:22:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:22:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:22:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:22:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:22:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:22:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:22:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:22:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:22:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:22:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:47.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:47.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:47.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:22:47.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:22:47.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:22:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:22:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:22:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:22:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:22:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:22:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:22:47.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:22:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:22:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:22:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:22:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:22:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:22:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:22:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:22:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:22:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:22:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:22:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:22:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:22:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:22:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:22:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:22:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:22:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:22:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:22:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:22:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:22:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:22:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:22:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:22:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:22:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:22:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:22:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:22:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:22:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:22:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:22:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:22:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:22:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:22:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:22:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:22:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:22:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:45
2024-05-08T01:22:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:22:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:22:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:22:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:47'
2024-05-08T01:22:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:22:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:22:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:22:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:22:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:22:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:22:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:22:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:49
2024-05-08T01:22:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:22:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:22:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:22:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:47'
2024-05-08T01:22:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:22:50.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:22:50.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:22:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:22:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:22:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:22:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:22:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:22:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:22:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:22:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:22:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:22:50.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:49'
2024-05-08T01:22:50.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:22:49'
2024-05-08T01:22:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:22:49'
2024-05-08T01:22:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:22:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:22:49</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:22:47</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:22:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:22:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:22:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:49'
2024-05-08T01:22:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:22:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:22:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:22:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:22:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:22:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:22:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:22:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:22:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:22:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:22:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:47
2024-05-08T01:22:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:22:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:22:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:22:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:49'
2024-05-08T01:22:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:22:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:22:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:22:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:22:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:22:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:22:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:22:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:22:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:22:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:22:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:22:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:22:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:47'
2024-05-08T01:22:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:22:47'
2024-05-08T01:22:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:22:47'
2024-05-08T01:22:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:22:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:22:47</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:22:49</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:22:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:22:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:22:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:47'
2024-05-08T01:22:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:22:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:22:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:22:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:22:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:22:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:22:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:22:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:22:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:22:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:22:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:22:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:22:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:22:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:22:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:22:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:22:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:22:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:22:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:22:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:22:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:22:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:22:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:22:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:22:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:22:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:22:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:22:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:22:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:22:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:22:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:22:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:22:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:22:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:22:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:22:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:22:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:22:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:22:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:22:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:22:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:22:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:22:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:22:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:22:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:22:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:22:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:22:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:22:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:22:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:22:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:22:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:22:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:22:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:22:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:22:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:22:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:22:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:22:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:22:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:22:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:22:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:22:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:22:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:22:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:22:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:22:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:22:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:22:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:22:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:22:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:22:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:22:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:22:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:22:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:22:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:22:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:22:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:22:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:22:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:22:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:22:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:22:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:22:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:22:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:22:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:22:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:22:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:22:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:22:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:22:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:22:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:22:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:22:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:22:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:22:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:22:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:53.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:22:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:22:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:22:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:22:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:22:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:22:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:22:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:22:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:22:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:22:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:22:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:22:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=64, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:22:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=64, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:22:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=64, entry_tid=0'
2024-05-08T01:22:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:22:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:22:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:22:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:22:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:22:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:22:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:22:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:22:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:22:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:22:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:22:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:22:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:22:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=64, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:22:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=64, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:22:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:22:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=64, entry_tid=0'
2024-05-08T01:22:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=64, entry_tid=0' -a 'object_number=0, tag_tid=64, entry_tid=0' = 'object_number=0, tag_tid=64, entry_tid=0'
2024-05-08T01:22:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:22:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:22:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:22:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:22:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:22:54.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:54.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:54.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:22:54.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:22:54.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:54.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:54
2024-05-08T01:22:54.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:54.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:22:54.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:22:54.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":61},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":64}}
2024-05-08T01:22:54.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:52'
2024-05-08T01:22:54.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:22:54.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:22:54.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:22:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:22:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:22:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:22:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:22:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:22:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:22:54.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:22:54.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:22:54.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:22:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:54'
2024-05-08T01:22:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:22:54'
2024-05-08T01:22:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:22:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:22:54'
2024-05-08T01:22:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:22:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:22:54</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:61},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:64}}</description><last_update>2024-05-08 01:22:52</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:22:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:22:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:22:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:54'
2024-05-08T01:22:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:22:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:22:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:22:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:22:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:22:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:22:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:22:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:22:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:22:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:22:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:22:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":61},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":64}}
2024-05-08T01:22:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:22:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:52
2024-05-08T01:22:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:22:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:22:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:22:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:22:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:54'
2024-05-08T01:22:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:22:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:22:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:22:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":61},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":64}}
2024-05-08T01:22:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:22:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:22:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:22:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:22:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:22:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:22:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:22:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:22:55.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:52'
2024-05-08T01:22:55.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:22:52'
2024-05-08T01:22:55.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:55.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:55.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:55.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:22:55.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:22:52'
2024-05-08T01:22:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:22:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:22:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:22:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:61},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:64}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:22:52</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:22:54</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:22:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:22:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:22:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:52'
2024-05-08T01:22:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:22:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:22:55.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:22:55.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:22:55.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:22:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:22:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:22:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:22:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:22:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:22:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:22:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:22:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:22:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:22:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:22:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:22:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:22:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:22:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:22:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:22:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:22:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:22:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:22:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:22:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:22:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:22:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:22:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:22:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:22:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:22:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:22:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:22:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:22:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:22:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:22:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:22:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:22:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:22:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:22:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:22:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:22:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:22:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:22:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:22:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:22:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:22:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:22:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:22:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:22:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:22:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:22:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:22:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:22:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:22:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:22:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:22:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:22:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:22:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:22:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:22:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:22:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:22:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:22:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:22:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:22:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:22:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:22:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:22:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:22:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:22:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:22:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:22:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:22:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:22:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:22:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:22:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:22:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:22:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:22:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:22:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:22:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:22:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:22:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:22:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:22:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:22:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:22:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:22:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:22:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:22:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:22:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:22:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:22:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:22:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:22:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:22:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:22:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:22:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:22:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:22:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:22:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:22:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:22:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:22:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:22:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:22:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:22:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:22:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:22:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:22:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:22:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:22:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:22:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:23:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:23:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:23:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:23:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:59
2024-05-08T01:23:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:23:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:23:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:23:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:58'
2024-05-08T01:23:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:23:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:23:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:23:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:23:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:23:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:23:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:23:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:23:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:23:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:23:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:59'
2024-05-08T01:23:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:22:59'
2024-05-08T01:23:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:22:59'
2024-05-08T01:23:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:23:00.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:22:59</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>stopped</description><last_update>2024-05-08 01:22:58</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:23:00.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:23:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:23:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:22:59'
2024-05-08T01:23:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:23:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:23:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:23:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:23:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:23:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:23:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:23:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:23:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:22:58
2024-05-08T01:23:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:23:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:23:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:23:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:59'
2024-05-08T01:23:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:23:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:23:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:23:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:23:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:03.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:03.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:23:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:23:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:23:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:02
2024-05-08T01:23:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:23:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:23:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:23:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:22:59'
2024-05-08T01:23:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:23:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:23:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:23:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:23:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:23:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:23:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:23:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:23:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:23:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:23:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:02'
2024-05-08T01:23:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:23:02'
2024-05-08T01:23:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:23:02'
2024-05-08T01:23:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:23:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:23:02</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:22:59</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:23:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:23:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:23:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:02'
2024-05-08T01:23:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:23:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:23:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:23:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:23:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:23:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:23:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:23:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:23:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:23:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:23:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:23:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:23:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:23:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:23:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:23:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:23:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:23:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:23:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:23:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:23:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:23:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:23:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:23:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:23:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:23:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:23:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:23:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:23:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:23:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:23:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:23:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:23:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:23:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:23:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:23:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:23:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:23:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:23:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:23:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:23:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:23:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:23:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:23:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:23:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:23:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:23:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:23:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:23:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:23:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:23:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:23:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:23:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:23:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:23:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:23:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:23:04.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:23:04.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:23:04.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:04.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:23:04.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:23:04.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:04.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:04.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:04.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:23:04.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:23:04.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:23:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:23:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:23:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:23:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:23:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:23:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:23:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:23:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:23:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:23:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:23:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:23:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:23:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:23:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:23:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:23:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:23:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:23:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:23:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:23:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:23:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:23:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:23:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:23:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:23:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:23:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:23:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:23:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:23:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:23:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:23:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:23:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:23:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:23:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:23:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:23:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:23:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:23:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:23:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:23:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:23:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:23:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:23:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:23:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:23:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:23:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:23:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:23:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:23:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:23:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:23:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:23:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=68, entry_tid=0'
2024-05-08T01:23:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:23:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:23:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:23:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:23:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:23:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:23:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:23:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:23:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:23:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:23:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:23:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:23:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=68, entry_tid=0'
2024-05-08T01:23:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=68, entry_tid=0' -a 'object_number=0, tag_tid=68, entry_tid=0' = 'object_number=0, tag_tid=68, entry_tid=0'
2024-05-08T01:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:23:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:23:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:23:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:23:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:23:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:02
2024-05-08T01:23:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:23:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:23:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":65},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":68}}
2024-05-08T01:23:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:04'
2024-05-08T01:23:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:23:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:23:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:23:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:23:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:23:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:23:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:23:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:23:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:07
2024-05-08T01:23:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":65},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":68}}
2024-05-08T01:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:04'
2024-05-08T01:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:23:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:23:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:23:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:23:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:23:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:23:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:23:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:23:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:23:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:23:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:23:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:07'
2024-05-08T01:23:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:23:07'
2024-05-08T01:23:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:23:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:23:07'
2024-05-08T01:23:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:23:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:23:07</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:65},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:68}}</description><last_update>2024-05-08 01:23:04</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:23:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:23:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:23:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:07'
2024-05-08T01:23:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:23:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:23:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:23:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:23:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:23:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:23:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:23:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:23:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:23:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:23:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:23:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":65},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":68}}
2024-05-08T01:23:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:04
2024-05-08T01:23:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:23:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:23:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:23:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:07'
2024-05-08T01:23:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:23:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:23:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:23:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":65},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":68}}
2024-05-08T01:23:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:23:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:23:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:23:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:23:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:23:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:23:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:23:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:23:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:04'
2024-05-08T01:23:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:23:04'
2024-05-08T01:23:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:23:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:23:04'
2024-05-08T01:23:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:23:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:65},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:68}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:23:04</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:23:07</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:23:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:23:09.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:23:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:04'
2024-05-08T01:23:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:23:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:23:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:23:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:23:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:23:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:23:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:23:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:23:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:23:09.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:23:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:23:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:23:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:23:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:23:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:23:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:23:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:23:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:23:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:23:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:23:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:23:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:23:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:23:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:23:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:23:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:23:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:23:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:23:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:23:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:23:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:23:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:23:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:23:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:23:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:23:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:23:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:23:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:23:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:23:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:23:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:23:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:23:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:23:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:23:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:23:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:23:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:23:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:23:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:23:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:23:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:23:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:23:10.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:10.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:10.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:10.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:23:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:23:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:23:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:23:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:23:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:23:11.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:23:11.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:23:11.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:11.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:23:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:23:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:23:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:23:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:23:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:23:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:23:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:23:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:23:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:23:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:23:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:23:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:23:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:23:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:23:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:23:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:23:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:23:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:23:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:23:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:23:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:23:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:23:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:23:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:23:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:23:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:23:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:23:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:23:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:23:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:23:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:23:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:23:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:23:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:23:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:23:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:23:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:23:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:23:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:23:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:23:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:23:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:23:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:23:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:23:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:23:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:23:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:23:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:14.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:23:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:23:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:23:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:14
2024-05-08T01:23:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:23:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:23:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:23:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:12'
2024-05-08T01:23:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:23:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:23:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:23:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:23:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:23:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:23:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:23:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:23:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:23:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:23:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:14'
2024-05-08T01:23:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:23:14'
2024-05-08T01:23:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:23:14'
2024-05-08T01:23:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:23:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:23:14</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:23:12</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:23:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:23:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:23:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:14'
2024-05-08T01:23:14.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:23:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:23:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:23:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:23:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:23:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:23:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:23:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:23:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:12
2024-05-08T01:23:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:23:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:23:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:23:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:14'
2024-05-08T01:23:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:23:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:23:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:23:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:23:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:23:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:23:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:23:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:23:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:23:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:23:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:12'
2024-05-08T01:23:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:23:12'
2024-05-08T01:23:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:23:12'
2024-05-08T01:23:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:23:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:23:12</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:23:14</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:23:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:23:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:23:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:12'
2024-05-08T01:23:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:23:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:23:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:23:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:23:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:23:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:23:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:23:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:23:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:23:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:23:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:23:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:23:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:23:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:23:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:23:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:23:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:23:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:23:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:23:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:23:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:23:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:23:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:23:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:23:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:23:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:23:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:23:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:23:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:23:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:23:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:23:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:23:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:23:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:23:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:23:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:23:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:23:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:23:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:23:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:23:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:23:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:23:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:23:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:23:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:23:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:23:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:23:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:23:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:23:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:23:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:23:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:23:16.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:23:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:23:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:23:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:23:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:23:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:23:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:23:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:23:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:23:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:23:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:23:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:23:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:23:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:23:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:23:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:23:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:23:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:23:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:23:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:23:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:23:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:23:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:23:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:23:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:23:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:23:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:23:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:23:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:23:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:23:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:23:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:23:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:23:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:23:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:23:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:23:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:23:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:23:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:23:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:23:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:23:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:23:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:23:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:23:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:23:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:23:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:23:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:23:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:23:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:23:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:23:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:23:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:23:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:23:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:23:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:23:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:23:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:23:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:23:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:23:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:23:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:23:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:23:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:23:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:23:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:23:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:23:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:23:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:23:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:23:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:23:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:23:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:23:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:23:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:23:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:23:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:23:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:23:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:23:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:23:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:23:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:23:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:23:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:23:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:23:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:23:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:23:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:23:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:23:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:23:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:23:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:23:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:23:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:23:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:23:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:23:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:23:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:23:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:23:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:23:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:23:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:23:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:23:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:23:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:23:19.338 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:23:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:23:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:23:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:23:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:23:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:23:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:23:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:23:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:23:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:23:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:23:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:23:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=69, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:23:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=69, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:23:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=69, entry_tid=0'
2024-05-08T01:23:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:23:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:23:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:23:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:23:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:23:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:23:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:23:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:23:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:23:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:23:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:23:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:23:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=69, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:23:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=69, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:23:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:19.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=69, entry_tid=0'
2024-05-08T01:23:19.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=69, entry_tid=0' -a 'object_number=0, tag_tid=69, entry_tid=0' = 'object_number=0, tag_tid=69, entry_tid=0'
2024-05-08T01:23:19.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:19.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:23:19.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:19.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:19.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:19.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:23:19.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:19.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:19.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:19.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:23:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:23:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:23:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:23:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:23:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:19
2024-05-08T01:23:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:23:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:23:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":66},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69}}
2024-05-08T01:23:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:17'
2024-05-08T01:23:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:23:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:23:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:23:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:23:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:23:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:23:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:23:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:23:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:23:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:23:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:23:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:23:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:19'
2024-05-08T01:23:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:23:19'
2024-05-08T01:23:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:23:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:23:19'
2024-05-08T01:23:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:23:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:23:19</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:66},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:69}}</description><last_update>2024-05-08 01:23:17</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:23:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:23:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:23:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:19'
2024-05-08T01:23:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:23:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:23:20.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:23:20.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:23:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:23:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:23:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:23:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:23:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:23:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:23:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:23:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":66},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69}}
2024-05-08T01:23:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:17
2024-05-08T01:23:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:23:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:23:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:23:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:19'
2024-05-08T01:23:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:23:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:23:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:23:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":66},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69}}
2024-05-08T01:23:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:23:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:23:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:23:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:23:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:23:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:23:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:23:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:23:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:17'
2024-05-08T01:23:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:23:17'
2024-05-08T01:23:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:23:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:23:17'
2024-05-08T01:23:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:23:21.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:66},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:69}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:23:17</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:23:19</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:23:21.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:23:21.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:23:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:17'
2024-05-08T01:23:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:23:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:23:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:23:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:23:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:23:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:23:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:23:21.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:23:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:23:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:23:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:23:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:23:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:22.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:23:22.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:22.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:22.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:22.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:23:22.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:23:22.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:23:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:23:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:23:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:23:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:23:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:23:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:23:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:23:22.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:23:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:23:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:23:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:23:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:23:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:23:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:23:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:23:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:23:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:23:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:23:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:23:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:23:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:23:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:23:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:23:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:23:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:23:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:23:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:23:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:23:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:23:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:23:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:23:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:23:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:23:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:23:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:23:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:23:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:23:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:23:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:23:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:23:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:23:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:23:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:23:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:23:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:23:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:23:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:23:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:19
2024-05-08T01:23:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:23:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:23:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:23:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:23'
2024-05-08T01:23:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:23:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:23:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:23:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:23:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:23:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:23:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:23:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:24
2024-05-08T01:23:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:23:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:23:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:23:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:23'
2024-05-08T01:23:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:23:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:23:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:23:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:23:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:23:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:23:26.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:26.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:23:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:23:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:23:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:23:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:24'
2024-05-08T01:23:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:23:24'
2024-05-08T01:23:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:23:24'
2024-05-08T01:23:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:23:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:23:24</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>stopped</description><last_update>2024-05-08 01:23:23</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:23:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:23:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:23:26.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:24'
2024-05-08T01:23:26.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:23:26.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:23:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:23:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:23:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:23:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:23:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:23:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:23:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:27
2024-05-08T01:23:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:23:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:23:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:23:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:24'
2024-05-08T01:23:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:23:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:23:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:23:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:23:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:23:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:23:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:23:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:23:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:23:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:23:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:27'
2024-05-08T01:23:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:23:27'
2024-05-08T01:23:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:23:27'
2024-05-08T01:23:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:23:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:23:27</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:23:24</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:23:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:23:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:23:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:27'
2024-05-08T01:23:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:23:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:23:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:23:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:23:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:23:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:23:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:23:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:23:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:23:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:23:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:23:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:23:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:23:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:23:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:23:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:23:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:23:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:23:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:23:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:23:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:23:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:23:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:23:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:23:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:23:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:23:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:23:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:23:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:23:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:23:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:23:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:23:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:23:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:23:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:23:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:23:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:23:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:23:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:23:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:23:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:23:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:23:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:23:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:23:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:23:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:23:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:23:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:23:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:23:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:23:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:23:28.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:23:28.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:23:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:23:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:23:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:23:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:23:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:23:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:23:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:23:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:23:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:23:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:23:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:23:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:23:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:23:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:23:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:23:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:23:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:23:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:23:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:23:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:23:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:23:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:23:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:23:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:23:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:23:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:23:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:23:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:23:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:23:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:23:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:23:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:23:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:23:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:23:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:23:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:23:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:23:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:23:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:23:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:23:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:23:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:23:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:23:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:23:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:23:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:23:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:23:31.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:23:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:23:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:23:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:23:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:23:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:23:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:23:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:23:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:23:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:23:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:23:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:23:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:23:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:23:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:23:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:23:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:23:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:23:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:23:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:23:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:23:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:23:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:23:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:23:31.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:23:31.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:23:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:23:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:23:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:23:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:23:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:23:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:23:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:23:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:23:31.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:31.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:31.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:31.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:23:31.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:23:31.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:23:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:23:31.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:31.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:23:31.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:23:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:23:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:23:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:23:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:23:31.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:31.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=73, entry_tid=0'
2024-05-08T01:23:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:23:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:23:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:23:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:23:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:23:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:23:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:23:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:23:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:23:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:23:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:23:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:23:31.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=73, entry_tid=0'
2024-05-08T01:23:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=73, entry_tid=0' -a 'object_number=0, tag_tid=73, entry_tid=0' = 'object_number=0, tag_tid=73, entry_tid=0'
2024-05-08T01:23:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:23:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:23:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:23:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:23:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:23:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:23:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:23:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:32
2024-05-08T01:23:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:23:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:23:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":70},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":73}}
2024-05-08T01:23:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:29'
2024-05-08T01:23:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:23:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:23:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:23:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:23:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:23:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:23:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:23:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:23:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:23:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:23:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:23:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:23:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:32'
2024-05-08T01:23:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:23:32'
2024-05-08T01:23:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:23:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:23:32'
2024-05-08T01:23:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:23:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:23:32</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:70},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:73}}</description><last_update>2024-05-08 01:23:29</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:23:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:23:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:23:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:32'
2024-05-08T01:23:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:23:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:23:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:23:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:23:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:23:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:23:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:23:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:23:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:23:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:23:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:23:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":70},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":73}}
2024-05-08T01:23:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:29
2024-05-08T01:23:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:23:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:23:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:23:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:32'
2024-05-08T01:23:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:23:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:23:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:23:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":70},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":73}}
2024-05-08T01:23:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:23:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:23:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:23:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:23:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:23:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:23:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:23:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:23:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:29'
2024-05-08T01:23:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:23:29'
2024-05-08T01:23:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:23:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:23:29'
2024-05-08T01:23:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:23:33.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:70},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:73}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:23:29</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:23:32</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:23:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:23:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:23:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:29'
2024-05-08T01:23:33.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:23:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:23:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:23:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:23:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:23:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:23:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:23:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:23:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:23:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:23:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:23:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:23:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:23:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:23:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:23:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:23:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:23:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:23:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:23:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:23:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:23:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:23:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:23:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:23:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:23:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:23:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:23:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:23:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:23:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:23:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:23:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:23:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:23:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:23:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:23:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:23:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:23:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:23:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:23:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:23:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:23:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:23:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:23:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:23:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:23:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:23:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:23:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:23:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:23:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:23:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:23:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:23:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:23:35.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:23:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:23:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:23:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:23:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:23:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:23:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:23:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:23:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:23:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:23:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:23:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:23:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:23:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:23:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:23:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:23:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:23:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:23:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:23:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:23:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:23:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:23:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:23:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:23:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:23:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:23:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:23:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:23:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:23:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:23:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:23:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:23:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:23:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:23:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:23:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:23:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:23:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:23:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:23:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:23:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:23:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:23:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:23:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:23:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:23:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:23:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:23:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:23:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:23:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:23:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:23:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:23:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:23:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:23:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:23:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:23:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:23:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:23:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:35
2024-05-08T01:23:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:23:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:23:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:23:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:37'
2024-05-08T01:23:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:23:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:23:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:38.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:23:40.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:23:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:23:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:23:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:23:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:39
2024-05-08T01:23:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:23:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:23:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:23:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:37'
2024-05-08T01:23:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:23:40.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:23:40.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:23:40.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:23:40.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:23:40.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:40.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:23:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:23:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:23:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:23:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:23:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:39'
2024-05-08T01:23:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:23:39'
2024-05-08T01:23:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:23:39'
2024-05-08T01:23:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:40.536 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:23:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:23:39</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:23:37</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:23:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:23:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:23:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:39'
2024-05-08T01:23:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:23:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:23:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:23:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:23:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:23:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:23:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:23:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:23:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:37
2024-05-08T01:23:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:23:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:23:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:23:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:39'
2024-05-08T01:23:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:23:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:23:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:23:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:23:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:23:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:23:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:23:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:23:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:23:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:23:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:37'
2024-05-08T01:23:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:23:37'
2024-05-08T01:23:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:23:37'
2024-05-08T01:23:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:23:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:23:37</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:23:39</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:23:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:23:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:23:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:37'
2024-05-08T01:23:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:23:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:23:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:23:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:23:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:23:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:23:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:23:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:23:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:23:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:23:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:23:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:23:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:23:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:23:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:23:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:23:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:23:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:23:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:23:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:23:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:23:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:23:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:23:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:23:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:23:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:23:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:23:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:23:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:23:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:23:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:23:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:23:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:23:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:23:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:23:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:23:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:23:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:23:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:23:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:23:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:23:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:23:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:23:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:23:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:23:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:23:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:23:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:23:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:23:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:23:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:23:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:23:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:23:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:23:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:23:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:23:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:23:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:23:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:23:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:23:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:23:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:23:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:23:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:23:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:23:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:23:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:23:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:23:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:23:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:23:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:23:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:23:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:23:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:23:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:23:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:23:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:23:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:23:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:23:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:23:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:23:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:23:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:23:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:23:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:23:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:23:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:23:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:23:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:23:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:23:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:23:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:23:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:23:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:23:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:23:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:23:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:23:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:23:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:23:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:23:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:23:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:23:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:23:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:23:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:23:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:23:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:23:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:23:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:23:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=74, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:23:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=74, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:23:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=74, entry_tid=0'
2024-05-08T01:23:43.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:23:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:23:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:23:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:23:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:23:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:23:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:23:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:23:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:23:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:23:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:23:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:23:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=74, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:23:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=74, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:23:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=74, entry_tid=0'
2024-05-08T01:23:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=74, entry_tid=0' -a 'object_number=0, tag_tid=74, entry_tid=0' = 'object_number=0, tag_tid=74, entry_tid=0'
2024-05-08T01:23:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:23:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:23:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:44.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:23:44.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:44.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:44.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:44.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:23:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:23:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:23:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:23:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:44
2024-05-08T01:23:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:23:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:23:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":71},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":74}}
2024-05-08T01:23:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:42'
2024-05-08T01:23:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:23:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:23:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:23:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:23:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:23:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:23:45.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:23:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:23:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:23:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:23:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:23:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:23:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:44'
2024-05-08T01:23:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:23:44'
2024-05-08T01:23:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:23:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:23:44'
2024-05-08T01:23:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:23:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:23:44</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:71},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:74}}</description><last_update>2024-05-08 01:23:42</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:23:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:23:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:23:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:44'
2024-05-08T01:23:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:23:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:23:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:23:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:23:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:23:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:23:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:23:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:23:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:23:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:23:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:23:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":71},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":74}}
2024-05-08T01:23:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:42
2024-05-08T01:23:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:23:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:23:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:23:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:44'
2024-05-08T01:23:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:23:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:23:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:23:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":71},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":74}}
2024-05-08T01:23:46.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:23:46.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:23:46.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:23:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:23:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:23:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:23:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:23:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:23:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:42'
2024-05-08T01:23:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:23:42'
2024-05-08T01:23:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:23:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:23:42'
2024-05-08T01:23:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:23:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:71},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:74}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:23:42</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:23:44</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:23:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:23:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:23:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:42'
2024-05-08T01:23:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:23:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:23:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:23:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:23:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:23:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:23:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:23:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:23:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:23:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:23:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:23:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:23:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:23:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:23:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:23:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:23:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:23:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:23:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:23:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:23:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:23:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:23:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:23:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:23:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:23:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:23:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:23:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:23:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:23:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:23:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:23:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:23:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:23:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:23:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:23:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:23:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:23:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:23:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:23:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:23:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:23:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:23:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:23:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:23:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:23:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:23:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:23:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:23:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:23:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:23:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:23:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:23:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:23:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:23:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:23:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:23:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:23:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:23:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:23:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:23:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:23:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:23:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:23:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:23:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:23:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:23:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:23:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:23:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:23:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:23:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:23:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:23:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:23:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:23:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:23:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:23:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:23:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:23:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:23:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:23:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:23:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:23:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:23:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:23:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:23:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:23:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:23:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:23:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:23:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:23:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:23:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:23:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:23:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:23:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:23:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:23:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:23:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:23:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:23:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:23:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:23:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:23:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:23:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:23:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:23:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:23:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:23:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:23:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:23:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:23:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:23:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:23:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:23:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:49
2024-05-08T01:23:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:23:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:23:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:23:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:48'
2024-05-08T01:23:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:23:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:23:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:23:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:23:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:23:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:23:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:23:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:23:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:23:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:23:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:49'
2024-05-08T01:23:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:23:49'
2024-05-08T01:23:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:23:49'
2024-05-08T01:23:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:23:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:23:49</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>stopped</description><last_update>2024-05-08 01:23:48</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:23:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:23:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:23:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:49'
2024-05-08T01:23:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:23:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:23:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:23:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:23:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:23:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:23:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:23:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:23:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:48
2024-05-08T01:23:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:23:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:23:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:23:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:49'
2024-05-08T01:23:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:23:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:23:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:23:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:23:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:23:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:23:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:23:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:52
2024-05-08T01:23:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:23:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:23:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:23:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:49'
2024-05-08T01:23:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:23:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:23:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:23:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:23:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:23:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:23:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:23:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:23:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:23:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:23:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:52'
2024-05-08T01:23:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:23:52'
2024-05-08T01:23:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:23:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:23:52'
2024-05-08T01:23:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:23:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:23:52</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:23:49</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:23:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:23:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:23:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:52'
2024-05-08T01:23:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:23:54.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:23:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:23:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:23:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:23:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:23:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:23:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:23:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:23:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:23:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:23:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:23:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:23:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:23:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:23:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:23:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:23:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:23:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:23:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:23:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:23:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:23:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:23:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:23:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:23:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:23:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:23:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:23:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:23:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:23:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:23:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:23:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:23:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:23:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:23:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:23:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:23:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:23:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:23:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:23:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:23:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:23:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:23:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:23:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:23:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:23:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:23:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:23:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:23:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:23:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:23:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:23:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:23:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:23:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:23:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:23:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:23:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:23:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:23:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:23:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:23:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:23:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:23:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:23:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:23:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:23:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:23:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:23:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:23:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:23:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:23:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:23:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:23:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:23:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:23:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:23:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:23:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:23:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:23:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:23:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:23:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:23:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:23:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:23:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:23:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:23:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:23:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:23:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:23:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:23:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:23:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:23:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:23:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:23:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:23:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:23:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:23:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:23:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:23:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:23:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:23:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:23:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:23:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:23:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:23:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:23:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:23:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:23:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:23:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:23:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:23:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:23:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:23:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:23:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:23:55.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:23:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:23:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:23:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:23:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:23:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:23:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:23:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:23:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:23:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:23:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:23:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:23:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=78, entry_tid=0'
2024-05-08T01:23:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:23:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:23:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:23:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:23:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:23:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:23:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:23:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:23:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:23:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:23:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:23:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:23:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:23:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:23:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:23:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:23:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=78, entry_tid=0'
2024-05-08T01:23:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=78, entry_tid=0' -a 'object_number=0, tag_tid=78, entry_tid=0' = 'object_number=0, tag_tid=78, entry_tid=0'
2024-05-08T01:23:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:23:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:23:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:23:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:23:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:23:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:23:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:23:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:23:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:52
2024-05-08T01:23:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:23:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:23:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":75},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":78}}
2024-05-08T01:23:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:54'
2024-05-08T01:23:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:23:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:23:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:23:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:23:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:23:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:58.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:58.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:23:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:23:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:23:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:23:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:23:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:23:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:57
2024-05-08T01:23:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:23:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:23:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:23:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":75},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":78}}
2024-05-08T01:23:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:54'
2024-05-08T01:23:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:23:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:23:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:23:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:23:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:23:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:23:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:23:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:23:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:23:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:23:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:23:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:23:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:57'
2024-05-08T01:23:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:23:57'
2024-05-08T01:23:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:23:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:23:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:23:57'
2024-05-08T01:23:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:23:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:23:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:23:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:23:57</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:75},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:78}}</description><last_update>2024-05-08 01:23:54</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:23:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:23:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:23:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:57'
2024-05-08T01:23:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:23:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:23:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:23:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:23:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:23:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:23:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:23:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:23:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:23:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:23:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:23:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:23:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:23:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:23:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:24:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:24:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:24:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:24:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":75},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":78}}
2024-05-08T01:24:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:23:54
2024-05-08T01:24:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:24:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:24:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:24:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:23:57'
2024-05-08T01:24:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:24:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:24:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:24:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":75},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":78}}
2024-05-08T01:24:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:24:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:24:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:24:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:24:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:24:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:24:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:24:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:24:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:54'
2024-05-08T01:24:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:23:54'
2024-05-08T01:24:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:24:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:23:54'
2024-05-08T01:24:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:24:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:75},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:78}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:23:54</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:23:57</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:24:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:24:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:24:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:23:54'
2024-05-08T01:24:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:24:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:24:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:24:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:24:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:24:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:24:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:24:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:24:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:24:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:24:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:24:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:24:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:24:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:24:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:24:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:24:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:24:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:24:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:24:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:24:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:24:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:24:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:24:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:24:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:24:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:24:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:24:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:24:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:24:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:24:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:24:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:24:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:24:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:24:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:24:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:24:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:24:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:24:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:24:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:24:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:24:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:24:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:24:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:24:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:24:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:24:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:02.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:24:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:24:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:24:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:24:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:01
2024-05-08T01:24:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:24:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:24:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:24:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:02'
2024-05-08T01:24:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:24:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:24:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:24:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:24:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:24:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:24:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:24:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:04
2024-05-08T01:24:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:24:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:24:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:24:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:02'
2024-05-08T01:24:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:24:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:24:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:24:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:24:04.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:24:04.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:04.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:24:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:24:04.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:24:04.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:24:04.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:24:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:04'
2024-05-08T01:24:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:24:04'
2024-05-08T01:24:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:04'
2024-05-08T01:24:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:24:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:04</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:24:02</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:24:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:24:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:24:04.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:04'
2024-05-08T01:24:04.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:24:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:24:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:24:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:24:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:24:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:05.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:05.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:05.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:24:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:24:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:24:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:02
2024-05-08T01:24:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:24:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:24:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:24:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:04'
2024-05-08T01:24:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:24:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:24:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:24:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:24:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:24:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:24:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:24:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:24:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:24:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:24:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:02'
2024-05-08T01:24:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:24:02'
2024-05-08T01:24:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:02'
2024-05-08T01:24:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:24:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:02</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:24:04</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:24:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:24:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:24:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:02'
2024-05-08T01:24:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:24:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:24:06.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:06.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:24:06.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:24:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:24:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:24:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:24:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:24:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:24:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:24:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:24:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:24:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:24:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:24:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:24:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:24:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:24:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:24:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:24:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:24:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:24:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:24:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:24:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:24:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:24:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:24:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:24:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:24:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:24:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:24:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:24:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:24:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:24:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:24:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:24:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:24:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:24:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:24:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:24:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:24:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:24:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:24:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:24:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:24:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:24:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:24:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:24:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:24:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:24:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:24:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:24:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:24:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:24:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:24:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:24:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:24:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:24:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:24:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:24:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:24:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:24:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:24:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:24:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:24:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:24:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:24:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:24:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:24:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:24:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:24:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:24:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:24:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:24:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:24:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:24:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:24:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:24:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:24:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:24:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:24:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:24:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:24:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:24:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:24:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:24:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:24:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:24:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:24:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:24:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:24:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:24:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:24:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:24:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:24:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:24:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:24:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:24:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:24:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:24:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:24:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:24:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:24:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:24:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:24:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:24:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:24:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:24:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:24:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=79, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=76, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=79, entry_tid=0'
2024-05-08T01:24:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:24:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:24:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:24:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:24:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:24:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:24:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:24:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:24:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:24:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:24:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=79, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=76, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=76, entry_tid=0'
2024-05-08T01:24:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=79, entry_tid=0' -a 'object_number=0, tag_tid=79, entry_tid=0' = 'object_number=0, tag_tid=76, entry_tid=0'
2024-05-08T01:24:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=76, entry_tid=0' '!=' ''
2024-05-08T01:24:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:24:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=76, entry_tid=0' = ''
2024-05-08T01:24:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=0, tag_tid=76, entry_tid=0'
2024-05-08T01:24:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=79, entry_tid=0'
2024-05-08T01:24:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:24:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:24:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=79
2024-05-08T01:24:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=76, entry_tid=0'
2024-05-08T01:24:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:24:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:24:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=76
2024-05-08T01:24:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=79, entry_tid=0'
2024-05-08T01:24:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:24:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:24:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=0
2024-05-08T01:24:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=76, entry_tid=0'
2024-05-08T01:24:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:24:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:24:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=0
2024-05-08T01:24:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 79 = 76 -a 0 -le 0
2024-05-08T01:24:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:24:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:24:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:24:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:24:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:24:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:24:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:24:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:24:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:24:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:24:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:24:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:24:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:24:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:24:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:24:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:24:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:24:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:24:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:24:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:24:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:24:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:24:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:24:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:24:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:24:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:24:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:24:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:24:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:24:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:24:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:24:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:24:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:24:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:24:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:24:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:24:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:24:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:24:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:24:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:24:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:24:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:24:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:24:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:24:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:24:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:24:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:24:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:24:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=79, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=79, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:08.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=79, entry_tid=0'
2024-05-08T01:24:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:24:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:24:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:24:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:24:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:24:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:24:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:24:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:24:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:24:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:24:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=79, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=79, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=79, entry_tid=0'
2024-05-08T01:24:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=79, entry_tid=0' -a 'object_number=0, tag_tid=79, entry_tid=0' = 'object_number=0, tag_tid=79, entry_tid=0'
2024-05-08T01:24:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:24:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:08.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:08.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:24:08.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:08.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:08.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:08.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:24:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:24:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:24:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:24:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:24:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:09
2024-05-08T01:24:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:24:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:24:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":76},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":79}}
2024-05-08T01:24:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:07'
2024-05-08T01:24:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:24:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:24:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:24:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:24:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:24:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:24:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:24:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:24:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:24:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:24:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:24:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:24:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:09'
2024-05-08T01:24:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:24:09'
2024-05-08T01:24:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:24:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:09'
2024-05-08T01:24:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:24:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:09</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:76},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:79}}</description><last_update>2024-05-08 01:24:07</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:24:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:24:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:24:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:09'
2024-05-08T01:24:09.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:24:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:24:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:24:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:24:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:24:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:24:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:24:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:24:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:24:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:24:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:24:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":76},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":79}}
2024-05-08T01:24:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:07
2024-05-08T01:24:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:24:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:24:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:24:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:09'
2024-05-08T01:24:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:24:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:24:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:24:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":76},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":79}}
2024-05-08T01:24:10.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:24:10.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:24:10.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:24:10.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:24:10.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:24:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:24:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:24:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:24:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:07'
2024-05-08T01:24:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:24:07'
2024-05-08T01:24:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:24:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:07'
2024-05-08T01:24:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:24:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:76},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:79}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:07</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:24:09</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:24:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:24:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:24:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:07'
2024-05-08T01:24:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:24:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:24:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:24:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:24:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:24:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:24:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:24:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:24:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:24:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:24:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:24:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:24:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:24:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:24:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:24:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:24:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:24:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:24:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:24:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:24:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:24:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:24:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:24:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:24:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:24:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:24:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:24:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:24:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:24:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:24:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:24:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:24:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:24:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:24:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:24:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:24:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:24:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:24:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:24:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:24:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:24:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:24:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:24:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:24:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:24:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:24:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:24:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:24:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:24:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:24:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:24:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:24:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:24:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:24:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:12.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:24:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:24:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:24:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:24:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:24:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:24:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:24:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:24:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:24:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:24:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:24:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:24:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:24:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:24:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:24:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:24:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:24:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:24:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:24:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:24:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:24:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:24:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:24:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:24:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:24:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:24:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:24:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:24:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:24:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:24:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:24:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:24:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:24:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:24:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:24:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:24:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:24:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:24:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:24:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:24:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:24:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:24:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:24:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:24:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:24:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:24:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:14
2024-05-08T01:24:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:24:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:24:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:24:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:13'
2024-05-08T01:24:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:24:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:24:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:24:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:24:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:24:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:24:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:24:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:24:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:24:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:24:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:14'
2024-05-08T01:24:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:24:14'
2024-05-08T01:24:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:14'
2024-05-08T01:24:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:24:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:14</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>stopped</description><last_update>2024-05-08 01:24:13</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:24:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:24:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:24:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:14'
2024-05-08T01:24:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:24:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:24:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:24:15.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:24:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:24:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:24:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:24:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:24:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:13
2024-05-08T01:24:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:14'
2024-05-08T01:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:24:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:24:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:24:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:24:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:24:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:17
2024-05-08T01:24:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:24:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:24:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:24:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:14'
2024-05-08T01:24:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:24:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:24:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:24:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:24:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:24:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:24:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:24:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:24:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:24:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:24:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:17'
2024-05-08T01:24:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:24:17'
2024-05-08T01:24:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:17'
2024-05-08T01:24:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:24:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:17</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:24:14</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:24:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:24:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:24:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:17'
2024-05-08T01:24:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:24:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:24:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:24:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:24:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:24:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:24:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:24:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:24:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:24:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:24:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:24:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:24:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:24:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:24:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:24:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:24:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:24:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:24:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:24:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:24:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:24:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:24:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:24:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:24:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:24:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:24:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:24:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:24:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:24:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:24:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:24:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:24:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:24:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:24:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:24:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:24:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:24:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:24:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:24:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:24:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:24:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:24:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:24:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:24:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:24:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:24:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:24:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:24:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:24:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:24:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:24:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:24:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:24:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:24:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:24:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:24:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:24:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:24:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:24:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:24:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:24:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:24:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:24:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:24:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:24:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:24:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:24:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:24:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:24:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:24:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:24:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:24:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:24:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:24:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:24:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:24:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:24:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:24:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:24:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:24:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:24:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:24:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:24:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:24:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:24:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:24:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:24:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:24:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:24:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:24:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:24:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:24:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:24:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:24:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:24:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:24:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:24:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:24:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:24:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:24:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:24:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:24:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:24:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:24:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:24:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:20.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:24:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:24:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:24:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=83, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:24:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=80, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:24:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=83, entry_tid=0'
2024-05-08T01:24:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:24:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:24:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:24:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:24:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:24:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:24:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:24:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:24:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:24:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:24:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=83, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:24:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=80, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:24:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=80, entry_tid=0'
2024-05-08T01:24:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=83, entry_tid=0' -a 'object_number=0, tag_tid=83, entry_tid=0' = 'object_number=0, tag_tid=80, entry_tid=0'
2024-05-08T01:24:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=80, entry_tid=0' '!=' ''
2024-05-08T01:24:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:24:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=80, entry_tid=0' = ''
2024-05-08T01:24:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=0, tag_tid=80, entry_tid=0'
2024-05-08T01:24:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=83, entry_tid=0'
2024-05-08T01:24:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:24:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:24:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=83
2024-05-08T01:24:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=80, entry_tid=0'
2024-05-08T01:24:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:24:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:24:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=80
2024-05-08T01:24:20.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=83, entry_tid=0'
2024-05-08T01:24:20.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:24:20.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:24:20.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=0
2024-05-08T01:24:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=80, entry_tid=0'
2024-05-08T01:24:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:24:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:24:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=0
2024-05-08T01:24:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 83 = 80 -a 0 -le 0
2024-05-08T01:24:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:24:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:24:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:24:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:24:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:24:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:24:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:24:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:24:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:24:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:24:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:24:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:24:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:24:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:24:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:24:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:24:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:24:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:24:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:24:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:24:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:24:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:24:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:24:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:24:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:24:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:24:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:24:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:24:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:24:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:24:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:24:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:24:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:24:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:24:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:24:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:24:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:24:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:24:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:24:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:24:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:24:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:24:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:24:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:24:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:24:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:24:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:24:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:24:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=83, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:24:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=83, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:24:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:20.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=83, entry_tid=0'
2024-05-08T01:24:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:24:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:24:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:24:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:24:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:24:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:24:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:24:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:24:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:24:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:24:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=83, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:24:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=83, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:24:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=83, entry_tid=0'
2024-05-08T01:24:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=83, entry_tid=0' -a 'object_number=0, tag_tid=83, entry_tid=0' = 'object_number=0, tag_tid=83, entry_tid=0'
2024-05-08T01:24:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:24:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:24:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:24:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:24:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:24:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:24:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:24:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:17
2024-05-08T01:24:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:24:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:24:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":80},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83}}
2024-05-08T01:24:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:19'
2024-05-08T01:24:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:24:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:24:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:24:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:24:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:24:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:22
2024-05-08T01:24:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:24:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:24:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":80},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83}}
2024-05-08T01:24:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:19'
2024-05-08T01:24:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:24:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:24:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:24:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:24:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:24:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:24:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:24:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:24:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:24:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:24:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:24:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:24:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:22'
2024-05-08T01:24:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:24:22'
2024-05-08T01:24:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:24:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:22'
2024-05-08T01:24:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:24:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:22</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:80},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:83}}</description><last_update>2024-05-08 01:24:19</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:24:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:24:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:24:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:22'
2024-05-08T01:24:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:24:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:24:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:24:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:24:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:24:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:24:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:24:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:24:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:24:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:24:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:24:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":80},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83}}
2024-05-08T01:24:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:19
2024-05-08T01:24:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:24:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:24:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:24:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:22'
2024-05-08T01:24:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:24:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:24:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:24:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":80},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83}}
2024-05-08T01:24:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:24:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:24:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:24:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:24:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:24:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:24:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:24:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:24:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:19'
2024-05-08T01:24:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:24:19'
2024-05-08T01:24:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:24:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:19'
2024-05-08T01:24:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:24:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:80},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:83}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:19</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:24:22</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:24:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:24:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:24:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:19'
2024-05-08T01:24:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:24:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:24:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:24:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:24:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:24:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:24:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:24:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:24:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:24:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:24:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:24:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:24:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:24:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:24:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:24:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:24:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:24:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:24:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:24:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:24:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:24:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:24:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:24:26.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:24:26.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:24:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:24:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:24:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:24:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:24:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:24:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:24:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:24:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:24:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:24:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:24:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:24:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:24:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:24:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:24:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:24:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:24:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:24:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:24:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:24:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:24:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:24:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:24:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:24:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:24:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:24:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:24:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:24:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:24:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:24:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:24:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:24:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:24:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:24:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:24:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:24:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:24:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:24:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:24:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:24:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:24:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:24:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:24:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:24:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:24:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:24:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:24:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:24:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:24:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:24:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:24:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:24:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:24:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:24:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:24:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:24:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:24:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:24:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:24:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:24:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:24:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:24:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:24:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:24:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:24:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:24:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:24:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:24:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:24:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:24:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:24:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:24:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:24:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:24:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:29.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:29.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:29.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:29.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:24:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:24:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:24:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:29
2024-05-08T01:24:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:24:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:24:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:24:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:27'
2024-05-08T01:24:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:24:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:24:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:24:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:24:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:24:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:24:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:24:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:24:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:24:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:24:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:29'
2024-05-08T01:24:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:24:29'
2024-05-08T01:24:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:29'
2024-05-08T01:24:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:24:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:29</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:24:27</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:24:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:24:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:24:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:29'
2024-05-08T01:24:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:24:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:24:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:24:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:24:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:24:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:24:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:24:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:24:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:27
2024-05-08T01:24:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:24:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:24:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:24:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:29'
2024-05-08T01:24:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:24:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:24:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:24:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:24:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:24:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:24:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:24:31.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:24:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:24:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:24:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:27'
2024-05-08T01:24:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:24:27'
2024-05-08T01:24:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:27'
2024-05-08T01:24:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:24:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:27</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:24:29</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:24:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:24:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:24:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:27'
2024-05-08T01:24:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:24:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:24:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:24:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:24:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:24:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:24:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:24:31.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:24:31.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:31.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:31.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:31.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:24:31.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:31.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:24:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:24:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:24:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:24:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:24:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:24:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:24:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:24:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:24:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:24:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:24:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:24:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:24:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:24:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:24:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:24:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:24:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:24:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:24:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:24:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:24:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:24:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:24:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:24:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:24:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:24:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:24:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:24:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:24:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:24:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:24:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:24:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:24:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:24:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:24:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:24:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:24:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:24:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:24:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:24:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:24:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:24:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:24:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:24:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:24:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:24:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:24:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:24:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:24:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:24:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:24:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:24:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:24:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:24:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:24:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:24:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:24:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:24:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:24:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:24:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:24:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:24:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:24:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:24:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:24:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:24:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:24:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:24:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:24:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:24:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:24:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:24:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:24:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:24:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:24:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:24:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:24:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:24:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:24:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:24:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:24:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:24:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:24:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:24:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:24:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:24:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:24:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:24:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:24:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:24:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:24:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:24:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:24:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:24:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:24:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:24:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:24:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:24:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:24:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=84, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=81, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=84, entry_tid=0'
2024-05-08T01:24:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:24:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:24:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:24:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:24:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:24:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:24:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:24:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:24:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:24:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:24:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=84, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=81, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=81, entry_tid=0'
2024-05-08T01:24:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=84, entry_tid=0' -a 'object_number=0, tag_tid=84, entry_tid=0' = 'object_number=0, tag_tid=81, entry_tid=0'
2024-05-08T01:24:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=81, entry_tid=0' '!=' ''
2024-05-08T01:24:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:24:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=81, entry_tid=0' = ''
2024-05-08T01:24:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=0, tag_tid=81, entry_tid=0'
2024-05-08T01:24:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=84, entry_tid=0'
2024-05-08T01:24:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:24:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:24:32.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=84
2024-05-08T01:24:32.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=81, entry_tid=0'
2024-05-08T01:24:32.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:24:32.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:24:32.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=81
2024-05-08T01:24:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=84, entry_tid=0'
2024-05-08T01:24:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:24:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:24:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=0
2024-05-08T01:24:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=81, entry_tid=0'
2024-05-08T01:24:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:24:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:24:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=0
2024-05-08T01:24:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 84 = 81 -a 0 -le 0
2024-05-08T01:24:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:24:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:24:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:24:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:24:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:24:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:24:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:24:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:24:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:24:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:24:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:24:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:24:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:24:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:24:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:24:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:24:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:24:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:24:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:24:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:24:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:24:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:24:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:24:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:24:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:24:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:24:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:24:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:24:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:24:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:24:33.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:24:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:24:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:24:33.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:24:33.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:24:33.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:24:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:24:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:24:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:24:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:24:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=84, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=84, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=84, entry_tid=0'
2024-05-08T01:24:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:24:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:24:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:24:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:24:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:24:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:24:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:24:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:24:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:24:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:24:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=84, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=84, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=84, entry_tid=0'
2024-05-08T01:24:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=84, entry_tid=0' -a 'object_number=0, tag_tid=84, entry_tid=0' = 'object_number=0, tag_tid=84, entry_tid=0'
2024-05-08T01:24:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:24:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:24:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:24:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:24:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:24:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:24:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:24:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:34
2024-05-08T01:24:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:24:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:24:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":81},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":84}}
2024-05-08T01:24:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:32'
2024-05-08T01:24:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:24:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:24:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:24:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:24:34.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:24:34.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:24:34.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:24:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:24:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:24:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:24:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:24:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:24:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:34'
2024-05-08T01:24:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:24:34'
2024-05-08T01:24:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:24:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:34'
2024-05-08T01:24:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:24:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:34</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:81},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:84}}</description><last_update>2024-05-08 01:24:32</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:24:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:24:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:24:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:34'
2024-05-08T01:24:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:24:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:24:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:24:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:24:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:24:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:24:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:24:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:24:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:24:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:24:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:24:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":81},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":84}}
2024-05-08T01:24:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:32
2024-05-08T01:24:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:24:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:24:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:24:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:34'
2024-05-08T01:24:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:24:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:24:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:24:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":81},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":84}}
2024-05-08T01:24:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:24:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:24:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:24:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:24:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:24:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:24:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:24:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:24:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:32'
2024-05-08T01:24:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:24:32'
2024-05-08T01:24:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:24:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:32'
2024-05-08T01:24:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:24:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:81},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:84}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:32</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:24:34</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:24:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:24:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:24:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:32'
2024-05-08T01:24:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:24:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:24:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:24:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:24:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:24:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:24:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:24:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:24:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:24:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:24:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:24:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:24:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:24:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:24:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:24:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:24:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:24:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:24:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:24:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:24:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:24:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:24:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:24:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:24:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:24:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:24:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:24:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:24:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:24:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:24:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:24:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:24:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:24:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:24:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:24:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:24:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:24:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:24:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:24:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopping"'
2024-05-08T01:24:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopping"'
2024-05-08T01:24:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:24:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:24:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:24:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:24:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:24:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:24:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:24:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:34
2024-05-08T01:24:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:24:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:24:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:24:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:37'
2024-05-08T01:24:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:24:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:24:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:24:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:24:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:24:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:24:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:24:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:39
2024-05-08T01:24:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:24:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:24:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:24:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:37'
2024-05-08T01:24:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:24:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:24:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:24:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:24:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:24:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:24:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:24:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:24:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:24:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:24:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:39'
2024-05-08T01:24:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:24:39'
2024-05-08T01:24:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:39'
2024-05-08T01:24:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:24:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:39</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>stopped</description><last_update>2024-05-08 01:24:37</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:24:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:24:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:24:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:39'
2024-05-08T01:24:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:24:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:24:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:40.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:24:40.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:24:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:24:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:24:41.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:24:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:24:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:37
2024-05-08T01:24:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:24:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:24:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:24:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:39'
2024-05-08T01:24:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:24:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:24:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:24:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:24:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:24:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:24:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:24:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:42
2024-05-08T01:24:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:24:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:24:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:24:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:39'
2024-05-08T01:24:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:24:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:24:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:24:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:24:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:24:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:24:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:24:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:24:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:24:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:24:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:42'
2024-05-08T01:24:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:24:42'
2024-05-08T01:24:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:42'
2024-05-08T01:24:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:24:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:42</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:24:39</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:24:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:24:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:24:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:42'
2024-05-08T01:24:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:24:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:24:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:24:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:24:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:24:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:24:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:24:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:24:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:24:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:24:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:24:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:24:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:24:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:24:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:24:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:24:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:24:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:24:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:24:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:24:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:24:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:24:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:24:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:24:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:24:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:24:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:24:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:24:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:24:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:24:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:24:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:24:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:24:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:24:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:24:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:24:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:24:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:24:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:24:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:24:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:24:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:24:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:24:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:24:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:24:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:24:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:24:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:24:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:24:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:24:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:24:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:24:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:24:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:24:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:24:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:24:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:24:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:24:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:24:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:24:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:24:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:24:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:24:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:24:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:24:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:24:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:24:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:24:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:24:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:24:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:24:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:24:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:24:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:24:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:24:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:24:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:24:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:24:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:24:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:24:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:24:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:24:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:24:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:24:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:24:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:24:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:24:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:24:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:24:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:24:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:24:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:24:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:24:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:24:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:24:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:24:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:24:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:24:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:24:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:24:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:24:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:24:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:24:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:24:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:24:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:24:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:24:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:24:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=88, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:24:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=88, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:24:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=88, entry_tid=0'
2024-05-08T01:24:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:24:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:24:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:24:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:24:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:24:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:24:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:24:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:24:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:24:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:24:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=88, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:24:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=88, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:24:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=88, entry_tid=0'
2024-05-08T01:24:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=88, entry_tid=0' -a 'object_number=0, tag_tid=88, entry_tid=0' = 'object_number=0, tag_tid=88, entry_tid=0'
2024-05-08T01:24:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:24:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:24:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:24:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:24:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:24:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:24:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:24:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:42
2024-05-08T01:24:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:24:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:24:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":85},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88}}
2024-05-08T01:24:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:44'
2024-05-08T01:24:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:24:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:24:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:24:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:24:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:24:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:24:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:24:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:24:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:47
2024-05-08T01:24:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:24:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:24:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":85},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88}}
2024-05-08T01:24:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:44'
2024-05-08T01:24:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:24:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:24:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:24:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:24:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:24:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:24:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:24:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:24:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:24:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:24:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:24:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:24:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:47'
2024-05-08T01:24:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:24:47'
2024-05-08T01:24:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:24:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:47'
2024-05-08T01:24:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:24:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:47</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:85},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:88}}</description><last_update>2024-05-08 01:24:44</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:24:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:24:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:24:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:47'
2024-05-08T01:24:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:24:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:24:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:24:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:24:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:24:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:24:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:24:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:24:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:24:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:24:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:24:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":85},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88}}
2024-05-08T01:24:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:44
2024-05-08T01:24:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:24:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:24:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:24:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:47'
2024-05-08T01:24:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:24:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:24:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:24:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":85},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88}}
2024-05-08T01:24:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:24:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:24:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:24:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:24:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:24:49.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:24:49.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:24:49.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:24:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:44'
2024-05-08T01:24:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:24:44'
2024-05-08T01:24:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:24:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:44'
2024-05-08T01:24:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:24:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:85},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:88}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:44</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:24:47</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:24:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:24:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:24:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:44'
2024-05-08T01:24:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:24:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:24:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:24:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:24:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:24:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:24:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:24:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:24:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:24:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:24:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:24:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:24:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:24:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:50.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:24:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:24:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:24:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:24:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:24:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:24:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:24:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:24:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:24:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:24:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:24:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:24:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:24:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:24:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:24:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:24:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:24:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:24:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:24:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:24:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:24:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:24:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:24:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:24:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:24:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:24:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:24:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:24:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:24:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:24:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:24:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:24:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:24:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:24:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:24:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:24:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:24:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:24:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:24:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:24:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:24:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:24:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:24:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:24:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:24:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:24:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:24:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:24:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:24:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:24:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:24:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:24:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:24:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:24:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:24:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:24:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:24:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:24:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:24:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:24:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:24:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:24:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:24:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:24:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:24:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:24:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:24:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:24:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:24:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:24:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:24:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:24:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:24:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:24:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:24:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:24:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:24:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:24:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:24:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:24:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:24:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:24:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:24:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:24:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:24:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:24:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:24:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:24:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:24:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:24:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:54
2024-05-08T01:24:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:24:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:24:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:24:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:52'
2024-05-08T01:24:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:24:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:24:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:24:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:24:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:24:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:24:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:24:54.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:24:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:24:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:24:54.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:54'
2024-05-08T01:24:54.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:24:54'
2024-05-08T01:24:54.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:54.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:54.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:54'
2024-05-08T01:24:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:24:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:54</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:24:52</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:24:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:24:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:24:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:54'
2024-05-08T01:24:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:24:54.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:24:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:24:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:24:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:24:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:24:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:24:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:24:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:24:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:24:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:24:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:52
2024-05-08T01:24:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:24:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:24:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:24:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:24:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:54'
2024-05-08T01:24:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:24:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:24:55.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:24:55.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:24:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:24:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:24:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:24:55.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:24:55.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:24:55.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:24:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:52'
2024-05-08T01:24:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:24:52'
2024-05-08T01:24:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:24:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:52'
2024-05-08T01:24:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:24:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:24:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:24:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:52</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:24:54</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:24:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:24:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:24:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:52'
2024-05-08T01:24:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:24:55.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:24:55.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:24:55.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:24:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:24:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:24:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:24:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:24:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:24:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:24:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:24:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:24:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:24:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:24:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:24:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:24:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:24:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:24:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:24:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:24:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:24:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:24:56.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:56.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:24:56.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:24:56.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:24:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:24:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:24:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:24:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:24:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:24:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:24:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:24:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:24:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:24:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:24:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:24:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:24:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:24:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:24:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:24:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:24:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:24:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:24:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:24:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:24:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:24:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:24:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:24:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:24:57.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:24:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:24:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:24:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:24:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:24:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:24:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:24:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:24:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:24:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:24:59.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:24:59.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:24:59.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:24:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:24:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:24:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:24:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:24:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:24:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:24:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:24:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:24:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:24:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:24:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:24:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:24:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:24:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:24:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:24:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:24:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:24:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:24:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:24:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:24:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:24:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:24:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:24:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:24:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:24:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:24:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:24:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:24:59.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:59.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:59.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:59.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:24:59.130 INFO:tasks.ceph.cluster1.osd.2.smithi106.stderr:2024-05-08T01:24:59.154+0000 7fe4d4916640 -1 reset not still connected to 0x55e1ffe91a00
2024-05-08T01:24:59.130 INFO:tasks.ceph.cluster1.osd.2.smithi106.stderr:2024-05-08T01:24:59.154+0000 7fe4d4916640 -1 reset not still connected to 0x55e20037cf70
2024-05-08T01:24:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:24:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:24:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:24:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:24:59.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:24:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:24:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:24:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:24:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:24:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:24:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:24:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:24:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:24:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:24:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:24:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:24:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:24:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:24:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:24:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:24:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:24:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:24:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:24:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:24:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:24:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:24:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:24:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:24:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:24:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:24:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:24:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:24:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:24:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:24:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:24:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:24:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:24:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:24:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:24:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:24:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:24:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:24:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:24:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:24:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:24:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:24:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:24:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:24:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:24:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:24:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:24:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:24:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:24:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:24:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:24:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:24:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:24:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:24:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:24:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:24:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:24:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:24:59.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:24:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:24:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:24:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:24:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:24:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:24:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:24:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:24:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:24:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:24:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=89, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=89, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:59.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:59.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=89, entry_tid=0'
2024-05-08T01:24:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:24:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:24:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:24:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:24:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:24:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:24:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:24:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:24:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:24:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:24:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:24:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:24:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:24:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=89, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=89, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:24:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:24:59.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=89, entry_tid=0'
2024-05-08T01:24:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=89, entry_tid=0' -a 'object_number=0, tag_tid=89, entry_tid=0' = 'object_number=0, tag_tid=89, entry_tid=0'
2024-05-08T01:24:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:24:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:24:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:24:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:24:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:24:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:24:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:24:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:24:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:24:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:25:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:25:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:25:00.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:00.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:00.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:25:00.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:25:00.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:59
2024-05-08T01:25:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:25:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:25:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":86},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":89}}
2024-05-08T01:25:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:57'
2024-05-08T01:25:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:25:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:25:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:25:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:00.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:25:00.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:25:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:00.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:00.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:59'
2024-05-08T01:25:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:24:59'
2024-05-08T01:25:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:25:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:59'
2024-05-08T01:25:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:59</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:86},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:89}}</description><last_update>2024-05-08 01:24:57</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:59'
2024-05-08T01:25:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:25:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:25:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:25:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:25:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:25:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:25:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:01.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:25:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:25:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:25:01.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:25:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":86},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":89}}
2024-05-08T01:25:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:24:57
2024-05-08T01:25:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:25:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:25:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:25:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:24:59'
2024-05-08T01:25:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:25:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:25:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":86},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":89}}
2024-05-08T01:25:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:25:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:25:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:57'
2024-05-08T01:25:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:24:57'
2024-05-08T01:25:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:25:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:24:57'
2024-05-08T01:25:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:86},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:89}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:24:57</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:24:59</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:24:57'
2024-05-08T01:25:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:02.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:25:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:25:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:25:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:25:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:25:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:25:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:25:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:25:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:25:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:25:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:25:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:25:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:25:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:25:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:25:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:25:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:25:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:25:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:25:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:25:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:25:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:25:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:25:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:25:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:25:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:25:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:25:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:25:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:25:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:25:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:25:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:25:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:25:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:25:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:25:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:25:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:25:03.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:25:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:25:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:25:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:25:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:25:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:25:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:25:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:03.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:25:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:25:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:25:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:25:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:25:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:25:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:25:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:25:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:25:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:25:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:25:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:25:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:25:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:25:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:25:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:25:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:25:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:25:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:25:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:25:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:25:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:25:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:25:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:25:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:25:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:25:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:25:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:25:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:25:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:25:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:25:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:25:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:25:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:25:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:25:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:25:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:25:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:25:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:25:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:25:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:25:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:25:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:25:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:25:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:25:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:25:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:04
2024-05-08T01:25:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:25:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:25:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:25:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:04'
2024-05-08T01:25:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:25:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:25:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:25:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:04'
2024-05-08T01:25:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:25:04'
2024-05-08T01:25:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:25:04'
2024-05-08T01:25:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:25:04</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>stopped</description><last_update>2024-05-08 01:25:04</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:04'
2024-05-08T01:25:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:25:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:25:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:25:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:25:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:25:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:25:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:25:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:07
2024-05-08T01:25:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:25:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:25:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:25:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:04'
2024-05-08T01:25:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:25:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:25:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:25:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:07'
2024-05-08T01:25:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:25:07'
2024-05-08T01:25:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:25:07'
2024-05-08T01:25:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:25:07</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:25:04</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:07.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:07'
2024-05-08T01:25:07.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:25:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:25:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:25:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:25:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:25:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:25:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:25:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:25:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:25:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:25:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:25:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:25:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:25:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:25:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:25:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:25:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:25:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:25:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:25:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:25:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:25:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:25:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:25:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:25:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:25:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:25:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:25:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:25:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:25:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:25:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:25:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:25:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:25:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:25:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:25:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:25:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:25:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:25:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:25:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:25:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:25:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:25:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:25:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:25:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:25:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:25:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:25:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:25:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:25:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:25:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:25:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:25:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:25:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:25:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:25:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:25:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:25:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:25:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:25:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:25:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:25:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:25:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:25:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:25:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:25:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:25:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:25:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:25:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:25:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:25:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:25:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:25:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:25:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:25:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:25:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:25:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:25:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:25:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:25:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:25:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:25:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:25:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:25:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:25:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:25:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:25:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:25:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:25:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:25:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:25:11.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:25:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:25:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:25:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:25:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:25:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:25:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:25:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:25:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:25:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:25:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:25:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:25:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:25:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:25:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:25:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:25:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:25:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:25:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:25:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:25:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:25:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:25:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:25:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:25:11.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:25:11.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:25:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:25:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:25:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:25:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:25:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:25:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:25:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:25:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:25:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:25:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:25:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:25:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:25:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:25:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:25:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:25:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:25:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:25:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:25:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:25:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:25:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:25:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:25:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:25:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:25:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:25:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:25:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:25:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:25:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:25:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:25:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:25:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:25:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=93, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:25:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=93, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:25:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=93, entry_tid=0'
2024-05-08T01:25:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:25:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:25:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:25:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:25:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:25:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:25:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:25:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:25:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:25:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:25:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=93, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:25:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=93, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:25:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=93, entry_tid=0'
2024-05-08T01:25:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=93, entry_tid=0' -a 'object_number=0, tag_tid=93, entry_tid=0' = 'object_number=0, tag_tid=93, entry_tid=0'
2024-05-08T01:25:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:25:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:25:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:25:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:25:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:25:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:25:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:25:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:12
2024-05-08T01:25:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:25:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:25:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":90},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":93}}
2024-05-08T01:25:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:09'
2024-05-08T01:25:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:25:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:25:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:25:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:25:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:25:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:12.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:12'
2024-05-08T01:25:12.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:25:12'
2024-05-08T01:25:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:25:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:25:12'
2024-05-08T01:25:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:25:12</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:90},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:93}}</description><last_update>2024-05-08 01:25:09</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:12'
2024-05-08T01:25:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:25:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:25:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:25:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:25:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:25:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:25:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:25:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:25:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:25:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:25:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":90},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":93}}
2024-05-08T01:25:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:09
2024-05-08T01:25:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:25:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:25:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:25:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:12'
2024-05-08T01:25:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:25:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:25:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":90},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":93}}
2024-05-08T01:25:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:25:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:25:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:09'
2024-05-08T01:25:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:25:09'
2024-05-08T01:25:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:25:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:25:09'
2024-05-08T01:25:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:90},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:93}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:25:09</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:25:12</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:09'
2024-05-08T01:25:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:25:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:25:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:25:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:25:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:25:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:25:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:25:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:25:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:25:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:25:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:25:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:25:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:25:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:25:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:25:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:25:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:25:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:25:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:25:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:25:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:25:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:25:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:25:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:25:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:25:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:25:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:25:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:25:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:25:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:25:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:25:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:25:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:25:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:25:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:25:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:25:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:25:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:25:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:25:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:25:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:25:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:25:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:25:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:25:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:25:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:25:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:25:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:25:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:25:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:25:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:25:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:25:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:25:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:25:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:25:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:25:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:25:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:25:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:25:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:25:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:25:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:25:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:25:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:25:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:25:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:25:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:25:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:25:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:25:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:25:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:25:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:25:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:25:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:25:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:25:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:25:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:25:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:25:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:25:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:25:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:25:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:25:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:25:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:25:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:25:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:25:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:25:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:25:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:25:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:25:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:25:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:25:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:25:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:25:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:25:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:25:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:25:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:25:18.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:18.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:18.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:25:18.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:25:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:15
2024-05-08T01:25:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:25:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:25:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:25:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:17'
2024-05-08T01:25:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:25:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:25:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:25:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:25:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:25:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:25:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:25:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:19
2024-05-08T01:25:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:25:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:25:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:25:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:17'
2024-05-08T01:25:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:25:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:25:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:25:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:20.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:20.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:20.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:19'
2024-05-08T01:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:25:19'
2024-05-08T01:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:25:19'
2024-05-08T01:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:25:19</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:25:17</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:20.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:19'
2024-05-08T01:25:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:25:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:25:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:25:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:25:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:25:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:25:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:25:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:17
2024-05-08T01:25:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:25:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:25:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:25:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:19'
2024-05-08T01:25:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:25:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:25:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:25:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:17'
2024-05-08T01:25:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:25:17'
2024-05-08T01:25:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:25:17'
2024-05-08T01:25:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:25:17</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:25:19</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:17'
2024-05-08T01:25:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:25:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:25:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:21.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:21.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:25:21.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:21.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:21.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:21.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:25:21.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:25:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:25:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:25:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:25:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:25:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:25:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:22.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:25:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:25:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:25:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:25:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:25:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:25:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:25:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:25:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:25:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:25:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:25:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:25:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:25:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:25:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:25:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:25:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:25:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:25:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:25:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:25:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:25:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:25:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:25:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:25:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:25:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:25:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:25:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:25:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:25:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:25:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:25:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:25:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:25:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:25:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:25:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:25:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:25:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:25:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:25:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:25:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:25:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:25:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:25:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:25:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:25:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:25:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:25:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:25:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:25:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:25:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:25:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:25:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:25:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:25:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:25:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:25:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:25:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:25:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:25:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:25:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:25:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:25:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:25:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:25:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:25:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:25:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:25:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:25:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:25:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:25:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:25:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:25:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:25:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:25:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:25:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:25:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:25:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:25:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:25:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:25:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:25:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:25:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:25:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:25:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:25:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:25:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:25:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:25:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:25:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:25:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:25:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:25:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:25:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:25:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:25:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:25:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:25:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=94, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:25:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=94, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:25:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=94, entry_tid=0'
2024-05-08T01:25:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:25:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:25:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:25:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:25:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:25:23.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:25:23.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:23.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:25:23.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:25:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:25:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:25:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=94, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:25:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=94, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:25:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=94, entry_tid=0'
2024-05-08T01:25:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=94, entry_tid=0' -a 'object_number=0, tag_tid=94, entry_tid=0' = 'object_number=0, tag_tid=94, entry_tid=0'
2024-05-08T01:25:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:25:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:25:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:24.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:25:24.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:24.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:24.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:25:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:25:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:25:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:25:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:24
2024-05-08T01:25:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:25:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:25:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":91},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":94}}
2024-05-08T01:25:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:22'
2024-05-08T01:25:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:25:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:25:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:25:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:25:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:25:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:24'
2024-05-08T01:25:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:25:24'
2024-05-08T01:25:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:25:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:25:24'
2024-05-08T01:25:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:25:24</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:91},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:94}}</description><last_update>2024-05-08 01:25:22</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:24'
2024-05-08T01:25:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:25:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:25:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:25:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:25:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:25:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:25:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:25:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:25:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:25:25.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:25:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":91},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":94}}
2024-05-08T01:25:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:22
2024-05-08T01:25:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:25:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:25:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:25:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:24'
2024-05-08T01:25:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:25:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:25:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":91},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":94}}
2024-05-08T01:25:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:25:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:25:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:22'
2024-05-08T01:25:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:25:22'
2024-05-08T01:25:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:25:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:25:22'
2024-05-08T01:25:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:91},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:94}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:25:22</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:25:24</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:22'
2024-05-08T01:25:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:25:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:25:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:25:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:25:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:25:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:25:26.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:25:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:25:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:25:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:25:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:25:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:25:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:25:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:25:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:25:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:25:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:25:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:25:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:25:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:25:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:25:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:25:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:25:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:25:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:25:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:25:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:25:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:25:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:25:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:25:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:25:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:25:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:25:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:25:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:25:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:25:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:25:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:25:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:25:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:25:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:25:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:25:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:25:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:25:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:25:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:25:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:25:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:25:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:25:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:25:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:25:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:25:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:25:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:25:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:25:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:25:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:25:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:25:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:25:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:25:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:25:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:25:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:25:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:25:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:25:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:25:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:25:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:25:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:25:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:25:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:25:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:25:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:25:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:25:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:25:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:25:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:25:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:25:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:25:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:25:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:25:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:25:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:25:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:25:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:25:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:25:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:25:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:25:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:25:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:29
2024-05-08T01:25:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:25:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:25:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:25:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:28'
2024-05-08T01:25:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:25:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:25:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:25:30.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:30.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:30.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:30.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:30.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:29'
2024-05-08T01:25:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:25:29'
2024-05-08T01:25:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:25:29'
2024-05-08T01:25:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:30.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:30.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:25:29</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>stopped</description><last_update>2024-05-08 01:25:28</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:29'
2024-05-08T01:25:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:25:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:25:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:25:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:25:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:25:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:25:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:25:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:28
2024-05-08T01:25:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:25:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:25:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:25:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:29'
2024-05-08T01:25:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:25:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:25:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:25:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:25:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:25:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:25:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:25:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:32
2024-05-08T01:25:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:25:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:25:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:25:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:29'
2024-05-08T01:25:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:25:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:25:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:25:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:34.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:34.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:34.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:34.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:34.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:32'
2024-05-08T01:25:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:25:32'
2024-05-08T01:25:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:25:32'
2024-05-08T01:25:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:25:32</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:25:29</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:32'
2024-05-08T01:25:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:25:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:25:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:25:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:25:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:25:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:25:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:25:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:25:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:25:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:25:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:25:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:25:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:25:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:25:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:25:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:25:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:25:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:25:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:25:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:25:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:25:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:25:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:25:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:25:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:25:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:25:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:25:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:25:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:25:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:25:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:25:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:25:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:25:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:25:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:25:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:25:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:25:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:25:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:25:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:25:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:25:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:25:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:25:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:25:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:25:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:25:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:25:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:25:35.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:25:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:25:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:25:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:25:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:25:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:25:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:25:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:25:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:25:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:25:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:25:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:25:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:25:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:25:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:25:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:25:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:25:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:25:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:25:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:25:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:25:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:25:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:25:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:25:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:25:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:25:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:25:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:25:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:25:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:25:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:25:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:25:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:25:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:25:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:25:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:25:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:25:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:25:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:25:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:25:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:25:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:25:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:25:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:25:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:25:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:25:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:25:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:25:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:25:35.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:25:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:25:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:25:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:25:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:25:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:25:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:25:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:25:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:25:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:25:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=98, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:25:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=98, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:25:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=98, entry_tid=0'
2024-05-08T01:25:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:25:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:25:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:25:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:25:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:25:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:25:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:25:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:25:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:25:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:25:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=98, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:25:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=98, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:25:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=98, entry_tid=0'
2024-05-08T01:25:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=98, entry_tid=0' -a 'object_number=0, tag_tid=98, entry_tid=0' = 'object_number=0, tag_tid=98, entry_tid=0'
2024-05-08T01:25:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:25:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:25:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:25:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:25:36.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:25:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:25:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:25:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:32
2024-05-08T01:25:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:25:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:25:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":95},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":98}}
2024-05-08T01:25:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:34'
2024-05-08T01:25:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:25:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:25:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:25:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:25:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:25:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:25:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:25:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:25:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:37
2024-05-08T01:25:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:25:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:25:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":95},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":98}}
2024-05-08T01:25:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:34'
2024-05-08T01:25:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:25:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:25:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:25:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:25:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:25:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:37'
2024-05-08T01:25:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:25:37'
2024-05-08T01:25:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:25:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:25:37'
2024-05-08T01:25:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:25:37</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:95},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:98}}</description><last_update>2024-05-08 01:25:34</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:39.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:37'
2024-05-08T01:25:39.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:39.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:25:39.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:25:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:25:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:25:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:25:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:25:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:25:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:25:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:25:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:25:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":95},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":98}}
2024-05-08T01:25:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:34
2024-05-08T01:25:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:25:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:25:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:25:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:37'
2024-05-08T01:25:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:25:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:25:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":95},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":98}}
2024-05-08T01:25:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:25:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:25:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:34'
2024-05-08T01:25:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:25:34'
2024-05-08T01:25:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:25:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:25:34'
2024-05-08T01:25:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:95},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:98}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:25:34</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:25:37</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:34'
2024-05-08T01:25:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:25:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:25:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:25:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:25:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:25:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:25:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:25:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:25:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:25:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:25:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:25:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:25:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:25:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:25:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:25:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:25:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:25:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:25:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:25:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:25:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:25:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:25:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:25:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:25:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:25:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:25:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:25:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:25:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:25:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:25:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:25:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:25:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:25:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:25:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:25:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:25:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:25:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:25:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:25:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:25:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:25:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:25:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:25:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:25:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:25:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:25:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:25:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:25:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:25:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:25:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:25:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:25:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:25:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:25:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:25:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:25:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:25:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:25:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:25:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:25:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:25:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:25:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:25:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:25:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:25:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:25:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:25:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:25:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:25:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:25:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:25:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:25:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:25:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:25:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:25:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:25:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:25:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:25:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:25:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:25:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:25:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:25:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:25:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:25:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:25:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:25:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:25:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:25:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:25:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:25:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:25:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:25:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:25:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:25:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:25:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:25:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:25:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:25:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:25:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:25:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:44
2024-05-08T01:25:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:25:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:25:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:25:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:42'
2024-05-08T01:25:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:25:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:25:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:44.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:25:44.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:44.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:44.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:44'
2024-05-08T01:25:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:25:44'
2024-05-08T01:25:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:25:44'
2024-05-08T01:25:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:25:44</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:25:42</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:44'
2024-05-08T01:25:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:25:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:25:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:25:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:25:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:25:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:25:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:25:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:42
2024-05-08T01:25:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:25:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:25:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:25:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:44'
2024-05-08T01:25:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:25:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:25:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:25:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:42'
2024-05-08T01:25:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:25:42'
2024-05-08T01:25:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:25:42'
2024-05-08T01:25:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:25:42</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:25:44</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:42'
2024-05-08T01:25:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:25:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:25:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:25:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:25:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:25:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:25:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:25:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:25:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:25:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:25:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:25:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:25:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:25:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:25:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:25:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:25:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:25:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:25:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:25:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:25:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:25:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:25:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:25:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:25:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:25:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:25:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:25:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:25:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:25:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:25:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:25:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:25:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:25:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:25:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:25:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:25:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:25:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:25:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:25:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:25:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:25:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:25:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:25:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:25:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:25:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:25:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:25:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:25:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:25:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:25:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:25:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:25:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:25:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:25:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:25:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:25:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:25:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:25:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:25:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:25:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:25:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:25:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:25:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:25:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:25:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:25:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:25:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:25:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:25:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:25:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:25:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:25:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:25:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:25:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:25:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:25:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:25:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:25:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:25:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:25:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:25:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:25:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:25:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:25:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:25:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:25:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:25:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:25:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:25:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:25:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:25:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:25:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:25:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:25:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:25:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:25:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:25:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:25:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:25:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:47.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:47.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:47.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:25:47.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:25:47.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:25:47.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:25:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:25:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:25:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:25:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:25:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:25:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=96, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:25:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=99, entry_tid=0'
2024-05-08T01:25:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:25:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:25:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:25:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:25:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:25:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:25:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:25:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:25:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:25:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:25:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:25:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=96, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:25:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=96, entry_tid=0'
2024-05-08T01:25:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=99, entry_tid=0' -a 'object_number=0, tag_tid=99, entry_tid=0' = 'object_number=0, tag_tid=96, entry_tid=0'
2024-05-08T01:25:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=96, entry_tid=0' '!=' ''
2024-05-08T01:25:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:25:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=96, entry_tid=0' = ''
2024-05-08T01:25:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=0, tag_tid=96, entry_tid=0'
2024-05-08T01:25:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=99, entry_tid=0'
2024-05-08T01:25:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:25:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:25:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=99
2024-05-08T01:25:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=96, entry_tid=0'
2024-05-08T01:25:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:25:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:25:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=96
2024-05-08T01:25:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=99, entry_tid=0'
2024-05-08T01:25:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:25:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:25:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=0
2024-05-08T01:25:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=96, entry_tid=0'
2024-05-08T01:25:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:25:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:25:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=0
2024-05-08T01:25:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 99 = 96 -a 0 -le 0
2024-05-08T01:25:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:25:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:25:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:25:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:25:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:25:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:25:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:25:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:25:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:25:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:25:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:25:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:25:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:25:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:25:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:25:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:25:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:25:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:25:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:25:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:25:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:25:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:25:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:25:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:25:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:25:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:25:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:25:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:25:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:25:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:25:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:25:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:25:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:25:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:25:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:25:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:25:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:25:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:25:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:25:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:25:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:25:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:25:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:25:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:25:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:25:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:25:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:25:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:25:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:25:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=96, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:25:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=99, entry_tid=0'
2024-05-08T01:25:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:25:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:25:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:25:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:25:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:25:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:25:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:25:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:25:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:25:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:25:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:25:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=96, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:25:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=96, entry_tid=0'
2024-05-08T01:25:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=99, entry_tid=0' -a 'object_number=0, tag_tid=99, entry_tid=0' = 'object_number=0, tag_tid=96, entry_tid=0'
2024-05-08T01:25:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=96, entry_tid=0' '!=' 'object_number=0, tag_tid=96, entry_tid=0'
2024-05-08T01:25:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:25:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.4
2024-05-08T01:25:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:25:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:25:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:25:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:25:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:25:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:25:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:25:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:25:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:25:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:25:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:25:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:25:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:25:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:25:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:25:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:25:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:25:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:25:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:25:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:25:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:25:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:25:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:25:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:25:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:25:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:25:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:25:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:25:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:25:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:25:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:25:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:25:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:25:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:25:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:25:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:25:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:25:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:25:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:25:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:25:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:25:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:25:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:25:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:25:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:25:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:25:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:25:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=99, entry_tid=0'
2024-05-08T01:25:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:25:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:25:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:25:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:25:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:25:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:25:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:25:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:25:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:25:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:25:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:25:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:25:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:25:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:25:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:25:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=99, entry_tid=0'
2024-05-08T01:25:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=99, entry_tid=0' -a 'object_number=0, tag_tid=99, entry_tid=0' = 'object_number=0, tag_tid=99, entry_tid=0'
2024-05-08T01:25:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:25:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:25:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:25:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:25:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:50.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:50.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:25:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:25:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:25:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:49
2024-05-08T01:25:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:25:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:25:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":96},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":99}}
2024-05-08T01:25:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:47'
2024-05-08T01:25:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:25:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:25:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:25:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:25:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:25:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:49'
2024-05-08T01:25:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:25:49'
2024-05-08T01:25:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:25:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:25:49'
2024-05-08T01:25:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:25:49</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:96},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:99}}</description><last_update>2024-05-08 01:25:47</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:49'
2024-05-08T01:25:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:50.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:25:50.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:25:50.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:25:50.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:25:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:25:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:50.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:25:50.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:50.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:50.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:50.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:25:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:25:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:25:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:25:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":96},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":99}}
2024-05-08T01:25:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:47
2024-05-08T01:25:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:25:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:25:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:25:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:49'
2024-05-08T01:25:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:25:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:25:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":96},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":99}}
2024-05-08T01:25:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:25:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:25:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:47'
2024-05-08T01:25:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:25:47'
2024-05-08T01:25:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:25:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:25:47'
2024-05-08T01:25:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:96},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:99}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:25:47</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:25:49</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:47'
2024-05-08T01:25:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:25:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:25:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:25:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:25:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:51.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:51.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:25:51.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:51.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:51.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:51.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:25:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:25:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:25:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:25:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:25:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:25:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:51.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:25:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:25:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:25:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:25:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:25:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:25:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:25:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:25:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:25:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:25:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:25:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:25:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:25:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:25:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:25:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:25:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:25:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:25:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:25:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:25:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:25:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:25:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:25:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:25:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:25:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:25:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:25:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:25:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:25:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:25:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:25:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:25:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:25:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:25:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:25:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:25:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:25:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:25:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:25:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:25:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:25:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:25:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:25:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:25:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:25:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:25:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:25:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:25:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:25:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:25:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:25:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:25:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:25:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:25:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:25:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:25:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:25:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:25:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:25:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:25:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:25:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:25:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:25:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:25:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:25:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:25:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:25:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:25:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:25:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:25:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:25:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:25:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:25:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:25:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:25:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:25:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:25:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:25:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:25:54.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:54.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:25:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:25:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:25:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:25:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:25:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:25:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:25:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:25:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:25:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:25:55.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:55.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:25:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:25:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:54
2024-05-08T01:25:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:25:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:25:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:25:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:53'
2024-05-08T01:25:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:25:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:25:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:55.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:25:55.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:55.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:55.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:54'
2024-05-08T01:25:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:25:54'
2024-05-08T01:25:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:25:54'
2024-05-08T01:25:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:56.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:25:54</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>stopped</description><last_update>2024-05-08 01:25:53</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:56.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:56.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:54'
2024-05-08T01:25:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:25:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:25:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:25:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:25:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:25:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:25:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:25:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:25:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:25:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:25:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:25:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:25:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:57
2024-05-08T01:25:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:25:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:25:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:25:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:25:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:54'
2024-05-08T01:25:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:25:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:25:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:25:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:25:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:25:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:25:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:25:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:25:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:25:57.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:25:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:57'
2024-05-08T01:25:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:25:57'
2024-05-08T01:25:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:25:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:25:57'
2024-05-08T01:25:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:25:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:25:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:25:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:25:57</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:25:54</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:25:57.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:25:57.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:25:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:57'
2024-05-08T01:25:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:25:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:25:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:25:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:25:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:25:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:25:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:25:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:25:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:25:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:25:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:25:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:25:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:25:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:25:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:25:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:25:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:25:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:25:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:25:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:25:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:25:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:25:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:25:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:25:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:25:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:25:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:25:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:25:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:25:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:25:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:25:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:25:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:25:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:25:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:25:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:25:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:25:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:25:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:25:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:25:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:25:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:25:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:25:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:25:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:25:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:25:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:25:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:25:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:25:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:25:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:25:58.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:25:58.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:25:58.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:25:58.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:25:58.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:25:58.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:25:58.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:25:58.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:25:58.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:25:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:25:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:25:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:25:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:25:58.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:25:58.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:25:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:25:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:25:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:25:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:25:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:26:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:26:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:26:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:26:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:26:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:26:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:26:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:26:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:26:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:26:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:26:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:26:00.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:26:00.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:26:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:26:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:26:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:26:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:26:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:26:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:26:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:26:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:26:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:26:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:26:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:26:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:26:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:26:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:26:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:26:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:26:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:26:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:26:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:26:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:26:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:26:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:26:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:26:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:26:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:26:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:26:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:26:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:26:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:26:00.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:26:00.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:26:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:26:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:26:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:26:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:26:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:26:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:26:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:26:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:26:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:26:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:26:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:26:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:26:00.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:26:00.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:26:00.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:26:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:26:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:26:00.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:26:00.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:26:00.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:26:00.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:26:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:26:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:26:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:26:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:26:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:26:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:26:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:26:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:26:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:26:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:26:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:26:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:00.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:26:00.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:26:00.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:26:00.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:26:00.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:26:00.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:26:00.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:26:00.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:26:00.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:26:00.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:00.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:26:00.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:26:00.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:26:00.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:26:00.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:00.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:26:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:26:00.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:26:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:26:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:26:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:26:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:26:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:26:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:26:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:26:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:26:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:26:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=103, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:26:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=103, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:26:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:01.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=103, entry_tid=0'
2024-05-08T01:26:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:26:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:26:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:26:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:26:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:26:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:26:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:26:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:26:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:26:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:26:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=103, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:26:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=103, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:26:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=103, entry_tid=0'
2024-05-08T01:26:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=103, entry_tid=0' -a 'object_number=0, tag_tid=103, entry_tid=0' = 'object_number=0, tag_tid=103, entry_tid=0'
2024-05-08T01:26:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:26:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:26:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:26:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:26:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:26:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:26:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:26:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:02
2024-05-08T01:26:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:26:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:26:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":100},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":103}}
2024-05-08T01:26:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:25:59'
2024-05-08T01:26:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:26:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:26:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:26:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:26:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:26:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:26:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:26:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:26:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:26:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:26:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:26:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:26:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:02'
2024-05-08T01:26:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:26:02'
2024-05-08T01:26:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:26:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:26:02'
2024-05-08T01:26:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:26:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:26:02</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:100},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:103}}</description><last_update>2024-05-08 01:25:59</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:26:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:26:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:26:02.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:02'
2024-05-08T01:26:02.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:26:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:26:02.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:26:02.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:26:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:26:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:26:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:26:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:26:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:26:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:26:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:26:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":100},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":103}}
2024-05-08T01:26:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:25:59
2024-05-08T01:26:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:26:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:26:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:26:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:02'
2024-05-08T01:26:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:26:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:26:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:26:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":100},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":103}}
2024-05-08T01:26:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:26:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:26:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:26:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:26:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:26:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:26:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:26:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:26:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:59'
2024-05-08T01:26:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:25:59'
2024-05-08T01:26:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:26:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:25:59'
2024-05-08T01:26:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:26:03.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:100},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:103}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:25:59</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:26:02</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:26:03.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:26:03.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:26:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:25:59'
2024-05-08T01:26:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:26:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:26:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:26:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:26:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:26:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:26:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:26:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:26:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:26:03.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:26:03.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:03.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:26:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:26:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:26:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:26:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:26:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:26:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:26:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:26:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:26:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:26:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:26:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:26:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:26:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:26:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:26:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:26:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:26:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:26:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:26:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:26:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:26:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:26:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:26:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:26:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:26:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:26:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:26:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:26:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:26:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:26:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:26:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:26:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:26:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:26:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:26:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:26:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:26:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:26:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:26:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:26:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:26:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:26:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:26:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:26:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:26:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:26:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:26:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:26:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:26:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:26:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:26:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:26:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:26:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:26:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:26:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:26:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:26:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:26:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:26:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:26:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:26:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:26:06.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:06.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:26:06.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:26:06.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:26:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:26:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:26:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:26:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:26:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:26:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:26:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:26:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:26:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:26:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:26:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:26:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:26:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:26:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:26:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:26:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:26:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:26:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:26:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:26:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:26:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:26:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:26:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:26:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:26:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:26:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:26:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:26:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:26:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:26:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:26:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:26:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:26:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:26:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:26:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:26:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:26:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:26:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:26:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:26:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:04
2024-05-08T01:26:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:26:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:26:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:26:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:07'
2024-05-08T01:26:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:26:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:26:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:26:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:26:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:26:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:26:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:26:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:09
2024-05-08T01:26:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:26:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:26:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:26:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:07'
2024-05-08T01:26:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:26:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:26:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:26:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:26:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:26:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:26:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:26:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:26:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:26:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:26:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:26:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:26:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:09'
2024-05-08T01:26:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:26:09'
2024-05-08T01:26:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:26:09'
2024-05-08T01:26:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:26:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:26:09</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:26:07</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:26:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:26:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:26:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:09'
2024-05-08T01:26:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:26:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:26:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:26:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:26:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:26:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:26:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:26:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:11.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:11.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:26:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:26:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:26:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:07
2024-05-08T01:26:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:26:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:26:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:26:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:09'
2024-05-08T01:26:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:26:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:26:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:26:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:26:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:26:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:26:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:26:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:26:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:26:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:26:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:26:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:26:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:07'
2024-05-08T01:26:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:26:07'
2024-05-08T01:26:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:26:07'
2024-05-08T01:26:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:26:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:26:07</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:26:09</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:26:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:26:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:26:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:07'
2024-05-08T01:26:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:26:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:26:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:26:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:26:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:26:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:26:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:26:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:26:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:26:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:26:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:26:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:26:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:26:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:26:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:26:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:26:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:26:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:26:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:26:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:26:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:26:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:26:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:26:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:26:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:26:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:26:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:26:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:26:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:26:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:26:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:26:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:26:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:26:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:26:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:26:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:26:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:26:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:26:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:26:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:26:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:26:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:26:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:26:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:26:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:26:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:26:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:26:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:26:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:26:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:26:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:26:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:26:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:26:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:26:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:26:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:26:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:26:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:26:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:26:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:26:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:26:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:26:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:26:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:26:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:26:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:26:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:26:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:26:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:26:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:26:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:26:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:26:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:26:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:26:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:26:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:26:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:26:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:26:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:26:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:26:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:26:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:26:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:26:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:26:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:26:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:26:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:26:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:26:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:26:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:26:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:26:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:26:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:26:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:26:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:26:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:26:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:26:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:26:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:26:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:26:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:26:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:26:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:26:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:26:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:26:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:26:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:26:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:26:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:26:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:26:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:26:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:26:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:26:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:26:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:26:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:26:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:26:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:26:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:26:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:26:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:26:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:26:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:26:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:26:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=104, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:26:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=104, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:26:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=104, entry_tid=0'
2024-05-08T01:26:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:26:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:26:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:26:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:26:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:26:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:26:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:26:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:26:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:26:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:26:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:26:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:26:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=104, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:26:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=104, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:26:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=104, entry_tid=0'
2024-05-08T01:26:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=104, entry_tid=0' -a 'object_number=0, tag_tid=104, entry_tid=0' = 'object_number=0, tag_tid=104, entry_tid=0'
2024-05-08T01:26:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:26:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:13.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:26:13.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:13.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:13.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:13.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:26:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:26:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:26:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:26:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:26:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:14
2024-05-08T01:26:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:26:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:26:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":101},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":104}}
2024-05-08T01:26:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:12'
2024-05-08T01:26:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:26:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:26:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:26:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:26:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:26:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:26:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:26:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:26:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:26:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:26:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:26:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:26:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:14'
2024-05-08T01:26:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:26:14'
2024-05-08T01:26:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:26:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:26:14'
2024-05-08T01:26:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:26:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:26:14</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:101},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:104}}</description><last_update>2024-05-08 01:26:12</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:26:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:26:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:26:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:14'
2024-05-08T01:26:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:26:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:26:14.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:26:14.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:26:14.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:26:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:26:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:26:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":101},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":104}}
2024-05-08T01:26:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:12
2024-05-08T01:26:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:26:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:26:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:26:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:14'
2024-05-08T01:26:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:26:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:26:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:26:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":101},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":104}}
2024-05-08T01:26:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:26:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:26:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:26:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:26:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:26:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:26:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:26:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:26:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:12'
2024-05-08T01:26:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:26:12'
2024-05-08T01:26:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:26:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:26:12'
2024-05-08T01:26:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:26:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:101},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:104}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:26:12</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:26:14</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:26:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:26:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:26:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:12'
2024-05-08T01:26:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:26:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:26:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:26:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:26:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:26:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:26:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:26:15.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:26:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:26:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:26:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:26:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:26:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:26:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:26:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:26:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:26:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:26:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:26:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:26:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:26:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:26:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:26:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:26:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:26:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:26:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:26:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:26:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:26:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:26:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:26:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:26:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:26:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:26:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:26:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:26:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:26:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:26:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:26:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:26:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:26:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:26:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:26:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:26:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:26:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:26:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:26:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:26:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:26:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:26:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:26:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:26:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:26:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:26:16.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:26:16.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:26:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:26:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:26:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:26:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:26:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:26:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:26:18.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:26:18.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:18.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:18.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:18.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:26:18.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:26:18.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:26:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:26:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:26:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:26:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:26:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:26:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:26:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:26:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:26:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:26:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:26:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:26:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:26:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:26:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:26:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:26:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:26:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:26:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:26:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:26:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:26:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:26:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:26:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:26:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:26:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:26:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:26:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:26:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:26:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:26:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:26:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:26:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:26:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:26:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:26:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:26:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:26:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:26:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:26:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:26:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:26:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:26:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:26:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:26:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:19.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:19.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:19.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:19.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:19.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:26:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:26:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:26:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:26:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:19
2024-05-08T01:26:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:26:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:26:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:26:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:17'
2024-05-08T01:26:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:26:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:26:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:26:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:26:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:26:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:26:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:26:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:26:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:26:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:26:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:26:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:26:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:19'
2024-05-08T01:26:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:26:19'
2024-05-08T01:26:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:26:19'
2024-05-08T01:26:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:26:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:26:19</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>stopped</description><last_update>2024-05-08 01:26:17</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:26:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:26:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:26:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:19'
2024-05-08T01:26:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:26:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:26:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:26:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:26:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:26:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:26:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:21.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:26:21.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:21.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:21.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:26:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:26:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:26:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:17
2024-05-08T01:26:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:26:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:26:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:26:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:19'
2024-05-08T01:26:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:26:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:26:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:26:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:26:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:26:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:22
2024-05-08T01:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:19'
2024-05-08T01:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:26:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:26:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:26:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:26:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:26:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:26:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:26:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:26:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:26:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:26:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:26:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:26:23.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:22'
2024-05-08T01:26:23.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:26:22'
2024-05-08T01:26:23.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:23.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:23.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:23.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:23.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:26:22'
2024-05-08T01:26:23.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:23.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:23.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:26:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:26:22</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:26:19</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:26:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:26:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:26:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:22'
2024-05-08T01:26:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:26:23.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:26:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:26:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:26:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:26:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:26:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:26:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:26:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:26:23.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:26:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:26:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:26:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:26:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:26:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:26:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:26:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:26:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:26:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:26:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:26:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:26:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:26:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:26:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:26:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:26:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:26:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:26:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:26:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:26:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:26:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:26:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:26:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:26:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:26:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:26:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:26:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:26:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:26:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:26:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:26:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:26:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:26:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:26:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:26:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:26:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:26:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:26:24.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:26:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:26:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:26:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:26:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:26:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:26:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:26:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:26:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:26:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:26:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:26:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:26:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:26:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:26:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:26:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:26:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:26:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:26:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:26:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:26:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:26:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:26:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:26:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:26:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:26:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:26:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:26:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:26:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:26:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:26:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:26:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:26:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:26:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:26:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:26:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:26:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:26:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:26:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:26:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:26:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:26:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:26:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:26:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:26:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:26:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:26:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:26:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:26:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:26:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:26:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:26:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:26:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:26:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:26:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:26:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:26:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:26:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:26:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:26:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:26:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:26:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:26:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:26:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:26:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:26:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=108, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:26:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=108, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:26:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=108, entry_tid=0'
2024-05-08T01:26:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:26:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:26:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:26:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:26:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:26:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:26:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:26:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:26:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:26:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:26:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=108, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:26:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=108, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:26:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=108, entry_tid=0'
2024-05-08T01:26:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=108, entry_tid=0' -a 'object_number=0, tag_tid=108, entry_tid=0' = 'object_number=0, tag_tid=108, entry_tid=0'
2024-05-08T01:26:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:26:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:26:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:26:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:26:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:26:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:26:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:26:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:22
2024-05-08T01:26:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:26:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:26:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":105},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":108}}
2024-05-08T01:26:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:24'
2024-05-08T01:26:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:26:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:26:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:26:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:26:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:26:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:28.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:28.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:28.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:28.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:26:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:26:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:26:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:27
2024-05-08T01:26:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:26:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:26:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":105},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":108}}
2024-05-08T01:26:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:24'
2024-05-08T01:26:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:26:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:26:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:26:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:26:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:26:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:26:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:26:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:26:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:26:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:26:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:26:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:26:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:27'
2024-05-08T01:26:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:26:27'
2024-05-08T01:26:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:26:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:26:27'
2024-05-08T01:26:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:26:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:26:27</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:105},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:108}}</description><last_update>2024-05-08 01:26:24</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:26:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:26:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:26:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:27'
2024-05-08T01:26:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:26:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:26:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:26:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:26:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:26:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:26:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:26:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:26:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:26:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:26:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:26:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":105},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":108}}
2024-05-08T01:26:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:24
2024-05-08T01:26:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:26:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:26:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:26:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:27'
2024-05-08T01:26:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:26:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:26:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:26:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":105},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":108}}
2024-05-08T01:26:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:26:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:26:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:26:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:26:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:26:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:26:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:26:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:26:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:24'
2024-05-08T01:26:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:26:24'
2024-05-08T01:26:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:26:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:26:24'
2024-05-08T01:26:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:26:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:105},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:108}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:26:24</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:26:27</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:26:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:26:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:26:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:24'
2024-05-08T01:26:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:26:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:26:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:26:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:26:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:26:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:26:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:26:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:26:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:26:29.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:26:29.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:26:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:26:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:26:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:26:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:26:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:26:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:26:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:26:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:26:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:26:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:26:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:26:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:26:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:26:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:26:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:26:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:26:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:26:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:26:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:26:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:26:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:26:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:26:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:26:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:26:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:26:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:26:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:26:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:26:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:26:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:26:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:26:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:26:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:26:30.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:26:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:26:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:26:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:26:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:26:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:26:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:26:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:26:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:26:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:26:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:26:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:26:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:26:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:26:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:26:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:26:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:26:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:26:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:26:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:26:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:26:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:30
2024-05-08T01:26:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:26:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:26:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:26:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:27'
2024-05-08T01:26:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:26:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:26:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:26:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:26:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:26:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:26:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:26:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:30
2024-05-08T01:26:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:26:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:26:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:26:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:32'
2024-05-08T01:26:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:26:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:26:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 4
2024-05-08T01:26:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:26:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:26:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:26:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:26:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:34
2024-05-08T01:26:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:26:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:26:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:26:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:37'
2024-05-08T01:26:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:26:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:26:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:26:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:26:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:26:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:26:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:26:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:26:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:26:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:26:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:26:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:26:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:34'
2024-05-08T01:26:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:26:34'
2024-05-08T01:26:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:26:34'
2024-05-08T01:26:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:26:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:26:34</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:26:37</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:26:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:26:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:26:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:34'
2024-05-08T01:26:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:26:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:26:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:26:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:26:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:26:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:26:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:26:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:26:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:26:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:26:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:37
2024-05-08T01:26:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:26:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:26:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:26:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:39'
2024-05-08T01:26:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:26:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:26:39.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:26:39.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:26:39.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:26:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:26:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:26:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:26:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:26:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:26:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:26:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:26:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:37'
2024-05-08T01:26:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:26:37'
2024-05-08T01:26:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:26:37'
2024-05-08T01:26:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:26:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:26:37</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:26:39</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:26:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:26:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:26:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:37'
2024-05-08T01:26:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:26:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:26:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:26:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:26:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:26:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:26:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:26:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:26:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:26:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:26:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:26:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:26:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:26:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:26:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:26:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:26:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:26:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:26:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:26:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:26:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:26:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:40.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:26:40.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:26:40.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:26:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:26:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:26:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:26:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:26:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:26:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:26:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:26:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:26:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:26:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:26:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:26:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:26:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:26:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:26:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:26:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:26:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:26:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:26:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:26:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:26:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:26:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:26:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:26:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:26:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:26:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:26:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:26:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:26:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:26:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:26:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:26:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:26:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:26:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:26:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:26:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:26:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:26:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:26:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:26:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:26:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:26:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:26:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:26:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:26:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:26:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:26:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:26:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:26:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:26:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:26:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:26:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:26:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:26:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:26:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:26:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:26:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:26:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:26:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:26:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:26:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:26:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:26:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:26:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:26:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:26:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:26:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:26:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:26:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:26:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:26:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:26:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:26:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:26:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:26:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:26:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:26:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:26:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:26:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:26:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:26:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:26:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:26:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:26:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:26:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:26:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:26:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:26:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:26:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:26:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:26:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:26:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:26:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:26:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:26:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:26:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:26:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:26:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:26:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:26:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:26:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:26:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:26:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:26:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:26:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:26:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:26:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:26:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:26:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:26:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:26:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:26:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:26:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:26:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:26:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:26:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:26:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:26:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:26:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:26:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:26:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:26:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:26:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:26:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:26:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:26:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:26:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:26:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:26:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:26:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:26:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:26:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:26:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:26:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:26:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:26:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:26:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:26:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:26:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:26:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:26:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:26:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:26:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:26:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:26:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:26:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:26:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:26:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=109, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:26:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=109, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:26:43.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=109, entry_tid=0'
2024-05-08T01:26:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:26:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:26:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:26:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:26:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:26:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:26:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:26:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:26:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:26:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:26:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:26:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:26:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=109, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:26:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=109, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:26:43.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=109, entry_tid=0'
2024-05-08T01:26:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=109, entry_tid=0' -a 'object_number=0, tag_tid=109, entry_tid=0' = 'object_number=0, tag_tid=109, entry_tid=0'
2024-05-08T01:26:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:26:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:26:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:26:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:26:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:26:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:26:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:26:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:44
2024-05-08T01:26:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:26:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:26:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":106},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":109}}
2024-05-08T01:26:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:42'
2024-05-08T01:26:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:26:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:26:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:26:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:26:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:26:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:26:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:26:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:26:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:26:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:26:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:26:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:26:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:44'
2024-05-08T01:26:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:26:44'
2024-05-08T01:26:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:26:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:26:44'
2024-05-08T01:26:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:26:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:26:44</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:106},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:109}}</description><last_update>2024-05-08 01:26:42</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:26:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:26:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:26:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:44'
2024-05-08T01:26:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:26:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:26:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:26:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:26:44.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:26:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:26:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:26:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:26:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:26:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:26:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:26:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":106},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":109}}
2024-05-08T01:26:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:42
2024-05-08T01:26:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:26:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:26:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:26:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:44'
2024-05-08T01:26:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:26:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:26:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:26:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":106},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":109}}
2024-05-08T01:26:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:26:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:26:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:26:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:26:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:26:45.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:26:45.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:26:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:26:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:42'
2024-05-08T01:26:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:26:42'
2024-05-08T01:26:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:26:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:26:42'
2024-05-08T01:26:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:26:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:106},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:109}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:26:42</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:26:44</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:26:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:26:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:26:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:42'
2024-05-08T01:26:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:26:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:26:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:26:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:26:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:26:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:26:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:26:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:26:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:26:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:26:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:26:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:26:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:26:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:26:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:26:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:26:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:26:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:26:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:26:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:26:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:26:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:26:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:26:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:26:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:26:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:26:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:26:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:26:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:26:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:26:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:26:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:26:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:26:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:26:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:26:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:26:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:26:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:26:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:26:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:26:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:26:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:26:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:26:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:26:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:26:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:26:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:26:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:26:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:26:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:26:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:26:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:26:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:26:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:26:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:26:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:26:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:26:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:26:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:26:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:26:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:26:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:26:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:26:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:26:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:26:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:26:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:26:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:26:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:26:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:26:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:26:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:26:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:26:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:26:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:26:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:26:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:26:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:26:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:26:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:26:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:26:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:26:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:26:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:26:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:26:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:26:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:26:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:26:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:26:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:26:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:26:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:26:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:26:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:26:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:26:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:26:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:26:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:26:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:26:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:26:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:26:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:26:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:26:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:26:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:26:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:49
2024-05-08T01:26:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:26:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:26:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:26:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:47'
2024-05-08T01:26:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:26:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:26:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:26:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:26:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:26:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:26:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:26:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:26:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:26:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:26:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:26:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:26:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:49'
2024-05-08T01:26:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:26:49'
2024-05-08T01:26:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:26:49'
2024-05-08T01:26:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:26:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:26:49</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>stopped</description><last_update>2024-05-08 01:26:47</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:26:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:26:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:26:50.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:49'
2024-05-08T01:26:50.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:26:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:26:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:26:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:26:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:26:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:26:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:26:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:26:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:26:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:26:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:47
2024-05-08T01:26:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:26:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:26:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:26:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:49'
2024-05-08T01:26:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:26:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:26:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:26:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:26:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:26:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:26:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:26:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:52
2024-05-08T01:26:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:26:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:26:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:26:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:49'
2024-05-08T01:26:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:26:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:26:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:26:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:26:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:26:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:26:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:26:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:26:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:26:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:26:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:26:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:26:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:52'
2024-05-08T01:26:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:26:52'
2024-05-08T01:26:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:26:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:26:52'
2024-05-08T01:26:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:26:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:26:52</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:26:49</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:26:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:26:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:26:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:52'
2024-05-08T01:26:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:26:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:26:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:26:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:26:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:26:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:26:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:26:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:26:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:26:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:26:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:26:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:26:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:26:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:26:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:26:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:26:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:26:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:26:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:26:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:26:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:26:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:26:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:26:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:26:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:26:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:26:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:26:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:26:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:26:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:26:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:26:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:26:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:26:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:26:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:26:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:26:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:26:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:26:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:26:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:26:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:26:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:26:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:26:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:26:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:26:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:26:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:26:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:26:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:26:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:26:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:26:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:26:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:26:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:26:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:26:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:26:55.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:26:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:26:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:26:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:26:55.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:26:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:26:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:26:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:26:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:26:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:26:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:26:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:26:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:26:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:26:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:26:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:26:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:26:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:26:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:26:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:26:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:26:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:26:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:26:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:26:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:26:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:26:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:26:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:26:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:26:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:26:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:26:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:26:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:26:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:26:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:26:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:26:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:26:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:26:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:26:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:26:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:26:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:26:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:26:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:26:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:26:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:26:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:26:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:26:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:26:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:26:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:26:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:26:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:26:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:26:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:26:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:26:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:26:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:26:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:26:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:26:55.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:26:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:26:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:26:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:26:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:26:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:26:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:26:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:26:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:26:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:55.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:26:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:26:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:26:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=113, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:26:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=113, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:26:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=113, entry_tid=0'
2024-05-08T01:26:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:26:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:26:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:26:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:26:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:26:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:26:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:26:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:26:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:26:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:26:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:26:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:26:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:26:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=113, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:26:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=113, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:26:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:26:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=113, entry_tid=0'
2024-05-08T01:26:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=113, entry_tid=0' -a 'object_number=0, tag_tid=113, entry_tid=0' = 'object_number=0, tag_tid=113, entry_tid=0'
2024-05-08T01:26:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:26:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:26:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:26:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:26:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:26:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:26:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:26:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:52
2024-05-08T01:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":110},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":113}}
2024-05-08T01:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:54'
2024-05-08T01:26:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:26:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:26:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:26:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:26:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:26:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:26:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:26:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:26:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:57
2024-05-08T01:26:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:26:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:26:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":110},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":113}}
2024-05-08T01:26:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:54'
2024-05-08T01:26:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:26:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:26:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:26:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:26:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:26:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:26:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:26:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:26:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:26:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:26:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:26:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:26:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:57'
2024-05-08T01:26:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:26:57'
2024-05-08T01:26:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:26:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:26:57'
2024-05-08T01:26:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:26:58.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:26:57</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:110},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:113}}</description><last_update>2024-05-08 01:26:54</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:26:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:26:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:26:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:57'
2024-05-08T01:26:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:26:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:26:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:26:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:26:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:26:58.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:58.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:58.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:26:58.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:58.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:58.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:58.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:26:58.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:58.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:58.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:26:58.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:26:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:26:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:26:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:26:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:26:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:26:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:26:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:26:59.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:26:59.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":110},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":113}}
2024-05-08T01:26:59.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:26:59.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:26:54
2024-05-08T01:26:59.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:26:59.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:26:59.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:26:59.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:26:59.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:26:57'
2024-05-08T01:26:59.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:26:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:26:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:26:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":110},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":113}}
2024-05-08T01:26:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:26:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:26:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:26:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:26:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:26:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:26:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:26:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:26:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:54'
2024-05-08T01:26:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:26:54'
2024-05-08T01:26:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:26:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:26:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:26:54'
2024-05-08T01:26:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:26:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:26:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:26:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:110},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:113}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:26:54</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:26:57</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:26:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:26:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:26:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:26:54'
2024-05-08T01:26:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:26:59.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:26:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:26:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:26:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:26:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:26:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:26:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster2 mirror test
2024-05-08T01:26:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:26:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:26:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:26:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test
2024-05-08T01:27:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:27:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:27:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:27:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:27:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:27:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:01.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:27:01.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:27:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:27:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:27:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:27:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:27:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:27:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:27:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:27:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:27:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:27:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:27:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:27:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:27:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:01.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:01.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:01.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:01.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:27:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:27:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:27:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:27:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:27:01.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:27:01.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:27:01.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:27:01.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:01.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:27:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:27:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:27:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:27:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:27:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:27:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:27:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:27:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:27:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:27:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:27:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:27:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:27:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:27:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:27:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:27:03.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:27:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:27:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:27:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:27:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:27:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:27:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:27:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:27:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:27:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:27:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:27:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:27:04
2024-05-08T01:27:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:27:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:27:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:27:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:27:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:27:02'
2024-05-08T01:27:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:27:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:27:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:27:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:27:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:27:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:27:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:27:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:27:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:27:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:27:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:27:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:27:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:04'
2024-05-08T01:27:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:27:04'
2024-05-08T01:27:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:27:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:27:04'
2024-05-08T01:27:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:27:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:27:04</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:27:02</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:27:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:27:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:27:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:04'
2024-05-08T01:27:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:27:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:27:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:27:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:27:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:27:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:27:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:27:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:27:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:27:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:27:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:27:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:27:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:27:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:27:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:27:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:27:02
2024-05-08T01:27:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:27:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:27:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:27:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:27:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:27:04'
2024-05-08T01:27:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:27:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:27:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:27:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:27:05.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:27:05.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:27:05.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:27:05.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:27:05.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:27:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:27:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:27:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:27:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:02'
2024-05-08T01:27:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:27:02'
2024-05-08T01:27:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:27:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:27:02'
2024-05-08T01:27:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:27:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:27:02</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:27:04</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:27:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:27:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:27:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:02'
2024-05-08T01:27:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:27:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:27:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:27:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:27:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:27:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test
2024-05-08T01:27:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:27:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test
2024-05-08T01:27:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:27:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster2 mirror test
2024-05-08T01:27:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test started
2024-05-08T01:27:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:27:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:27:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:27:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:27:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:27:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:27:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:27:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:27:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:27:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:27:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:27:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:27:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:27:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:27:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:27:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:27:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:27:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:27:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:27:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:27:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:27:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:27:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:27:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:27:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:27:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:27:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:27:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:27:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:27:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:27:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:27:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:27:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:27:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:27:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:27:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:27:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test started
2024-05-08T01:27:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:27:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:27:09.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:27:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:27:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:27:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:27:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:27:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:27:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:27:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:27:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:27:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:27:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:27:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:27:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:27:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:27:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:27:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:27:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:27:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:27:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:27:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:27:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:27:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:27:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:27:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:27:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:27:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:27:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:27:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:27:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:27:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:27:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:27:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:27:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:27:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:27:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:27:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test
2024-05-08T01:27:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster2
2024-05-08T01:27:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:27:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:27:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:27:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:27:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:27:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster2 mirror test
2024-05-08T01:27:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:27:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:27:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:27:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test
2024-05-08T01:27:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:27:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:27:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:27:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:27:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:27:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:27:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test
2024-05-08T01:27:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:27:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:27:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:27:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:27:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:27:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:27:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:27:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:27:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:27:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:27:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:27:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:27:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:27:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:27:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:27:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:27:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:27:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:27:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:27:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:27:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:27:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:27:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:27:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster1 mirror test
2024-05-08T01:27:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:27:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test ''
2024-05-08T01:27:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:27:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:27:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:27:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:27:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:27:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:27:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:27:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:27:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:27:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:27:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:27:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:27:09.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:27:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:27:09.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=114, entry_tid=0'
2024-05-08T01:27:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster1 mirror test
2024-05-08T01:27:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:27:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster1 mirror test '..*'
2024-05-08T01:27:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:27:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:27:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:27:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:27:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:27:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:27:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 journal status --image mirror/test
2024-05-08T01:27:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:27:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:27:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:27:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:27:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:27:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=60884b4e-dce4-4abb-b49d-86c95c49f777, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected]
2024-05-08T01:27:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:27:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=114, entry_tid=0'
2024-05-08T01:27:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=114, entry_tid=0' -a 'object_number=0, tag_tid=114, entry_tid=0' = 'object_number=0, tag_tid=114, entry_tid=0'
2024-05-08T01:27:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped
2024-05-08T01:27:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:27:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' ''
2024-05-08T01:27:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:27:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:27:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:27:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:27:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:27:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:27:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:27:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:27:09
2024-05-08T01:27:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:27:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:27:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:27:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":111},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":114}}
2024-05-08T01:27:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:27:07'
2024-05-08T01:27:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:27:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:27:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:27:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:27:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:27:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:27:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:27:10.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:27:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:27:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:27:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:27:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:27:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:09'
2024-05-08T01:27:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-05-08 01:27:09'
2024-05-08T01:27:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:27:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:27:09'
2024-05-08T01:27:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:27:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:27:09</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:111},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:114}}</description><last_update>2024-05-08 01:27:07</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:27:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:27:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:27:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:09'
2024-05-08T01:27:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:27:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:27:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:27:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:27:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:27:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying
2024-05-08T01:27:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:27:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' ''
2024-05-08T01:27:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:27:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:27:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:27:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:27:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:27:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":111},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":114}}
2024-05-08T01:27:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:27:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:27:07
2024-05-08T01:27:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:27:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:27:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:27:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:27:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:27:09'
2024-05-08T01:27:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:27:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:27:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:27:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":111},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":114}}
2024-05-08T01:27:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:27:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:27:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:27:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:27:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:27:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:27:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:27:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:27:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:07'
2024-05-08T01:27:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-05-08 01:27:07'
2024-05-08T01:27:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:27:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:27:07'
2024-05-08T01:27:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:27:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:111},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:114}}</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:27:07</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:27:09</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:27:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:27:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:27:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:07'
2024-05-08T01:27:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:27:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:27:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:27:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:27:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:27:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror test
2024-05-08T01:27:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test
2024-05-08T01:27:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:27:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test
2024-05-08T01:27:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped
2024-05-08T01:27:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:27:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:27:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:27:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:27:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:27:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:27:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:27:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:27:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:27:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:27:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:27:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:27:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:27:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:27:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:27:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:27:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:27:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:27:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:27:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:27:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:27:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:27:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:27:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:27:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:27:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:27:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:27:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:27:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:27:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:13.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:13.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:13.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:27:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:27:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:27:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:27:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:27:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:27:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:27:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = started
2024-05-08T01:27:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:27:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test stopped
2024-05-08T01:27:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:27:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:27:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:27:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test
2024-05-08T01:27:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:27:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:27:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:27:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:27:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:27:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:27:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test
2024-05-08T01:27:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:27:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:27:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:27:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:27:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:27:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:27:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:27:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:27:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:27:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:27:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:27:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:27:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:27:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:27:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:27:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:27:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:27:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:27:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:27:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:27:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:27:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown
2024-05-08T01:27:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:27:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' ''
2024-05-08T01:27:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:27:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:27:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:27:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:27:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:27:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:27:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:27:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:27:14
2024-05-08T01:27:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:27:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:27:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:27:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: stopped
2024-05-08T01:27:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:27:13'
2024-05-08T01:27:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:27:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:27:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:27:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:27:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:27:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:27:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:27:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:27:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:27:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:27:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:27:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:27:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:14'
2024-05-08T01:27:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-05-08 01:27:14'
2024-05-08T01:27:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:27:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:27:14'
2024-05-08T01:27:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:27:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:27:14</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>stopped</description><last_update>2024-05-08 01:27:13</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:27:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:27:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:27:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:14'
2024-05-08T01:27:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:27:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:27:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:27:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:27:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:27:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown
2024-05-08T01:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' ''
2024-05-08T01:27:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:27:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:27:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:27:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:27:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+unknown
2024-05-08T01:27:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: remote image is not primary
2024-05-08T01:27:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:27:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:27:17
2024-05-08T01:27:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:27:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:27:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+unknown
2024-05-08T01:27:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: remote image is not primary
2024-05-08T01:27:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:27:14'
2024-05-08T01:27:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+unknown'
2024-05-08T01:27:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+unknown
2024-05-08T01:27:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:27:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: remote image is not primary
2024-05-08T01:27:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:27:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:27:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:27:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:27:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:27:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:27:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:27:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:27:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:17'
2024-05-08T01:27:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-05-08 01:27:17'
2024-05-08T01:27:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+unknown
2024-05-08T01:27:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:27:17'
2024-05-08T01:27:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:27:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+unknown</state><description>remote image is not primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:27:17</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+unknown</state><description>remote image is not primary</description><last_update>2024-05-08 01:27:14</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:27:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:27:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:27:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:17'
2024-05-08T01:27:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:27:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+unknown
2024-05-08T01:27:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+unknown
2024-05-08T01:27:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+unknown
2024-05-08T01:27:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+unknown
2024-05-08T01:27:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster2 mirror test
2024-05-08T01:27:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=
2024-05-08T01:27:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test
2024-05-08T01:27:17.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:27:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:27:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:27:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:27:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:27:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:27:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:27:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:27:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:27:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:27:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:27:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:27:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:27:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:27:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:27:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:27:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:27:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:27:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:27:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:27:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:27:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:27:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:27:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:27:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = stopped
2024-05-08T01:27:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:27:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:27:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:27:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:27:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:27:21.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:27:21.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:27:21.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:27:21.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:21.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:27:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:27:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:27:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:27:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:27:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:27:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:27:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:27:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:27:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:27:21.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:27:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:27:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:27:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:27:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:27:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:27:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:27:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:27:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test
2024-05-08T01:27:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:27:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:27:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:27:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:27:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:27:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:27:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test
2024-05-08T01:27:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:27:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:27:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test'
2024-05-08T01:27:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test
2024-05-08T01:27:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:27:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:27:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:27:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:27:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:27:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test
2024-05-08T01:27:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:27:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:27:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:27:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:27:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:27:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:27:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:27:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:27:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:27:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:27:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:27:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:27:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:27:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test
2024-05-08T01:27:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test
2024-05-08T01:27:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:27:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test ''
2024-05-08T01:27:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:27:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:27:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:27:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:27:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:27:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:27:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:27:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:27:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:27:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:27:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:27:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=118, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:27:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=118, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:27:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:27:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=0, tag_tid=118, entry_tid=0'
2024-05-08T01:27:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test
2024-05-08T01:27:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:27:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test '..*'
2024-05-08T01:27:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:27:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:27:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test.status
2024-05-08T01:27:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test.status
2024-05-08T01:27:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:27:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:27:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test
2024-05-08T01:27:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:27:21.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:27:21.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:27:21.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:27:21.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=0, tag_tid=118, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:27:21.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=118, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected]
2024-05-08T01:27:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test.status
2024-05-08T01:27:21.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=118, entry_tid=0'
2024-05-08T01:27:21.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=0, tag_tid=118, entry_tid=0' -a 'object_number=0, tag_tid=118, entry_tid=0' = 'object_number=0, tag_tid=118, entry_tid=0'
2024-05-08T01:27:21.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:21.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped
2024-05-08T01:27:21.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:21.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:27:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' ''
2024-05-08T01:27:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:27:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test
2024-05-08T01:27:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:27:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:27:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:27:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:27:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:27:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:27:22
2024-05-08T01:27:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:27:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:27:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:27:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}
2024-05-08T01:27:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:27:19'
2024-05-08T01:27:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:27:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:27:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:27:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:27:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:27:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:27:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:27:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:27:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:27:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:27:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:27:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:27:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:22'
2024-05-08T01:27:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-05-08 01:27:22'
2024-05-08T01:27:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:27:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:27:22'
2024-05-08T01:27:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:27:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:27:22</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:115},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:118}}</description><last_update>2024-05-08 01:27:19</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:27:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:27:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:27:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:22'
2024-05-08T01:27:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:27:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:27:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:27:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:27:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:27:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying
2024-05-08T01:27:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:27:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' ''
2024-05-08T01:27:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:27:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:27:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:27:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   915c10af-6bd7-44b0-837b-a3e596bc8472
2024-05-08T01:27:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:27:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}
2024-05-08T01:27:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:27:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:27:19
2024-05-08T01:27:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:27:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:27:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:27:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:27:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:27:22'
2024-05-08T01:27:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:27:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:27:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:27:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}
2024-05-08T01:27:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:27:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:27:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:27:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:27:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:27:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:27:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:27:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:27:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:19'
2024-05-08T01:27:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:27:19'
2024-05-08T01:27:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:27:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:27:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:27:19'
2024-05-08T01:27:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:27:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:115},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:118}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:27:19</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:27:22</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:27:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:27:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:27:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:19'
2024-05-08T01:27:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:27:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:27:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:27:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:27:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:27:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = snapshot ']'
2024-05-08T01:27:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: force promote'
2024-05-08T01:27:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:27:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:27:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:27:24 TEST: force promote
2024-05-08T01:27:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:27:24 TEST: force promote
2024-05-08T01:27:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ force_promote_image=test_force_promote
2024-05-08T01:27:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image_and_enable_mirror cluster2 mirror test_force_promote journal
2024-05-08T01:27:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:27:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n journal ']'
2024-05-08T01:27:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image cluster2 mirror test_force_promote
2024-05-08T01:27:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=128
2024-05-08T01:27:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:27:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test_force_promote
2024-05-08T01:27:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ is_pool_mirror_mode_image mirror
2024-05-08T01:27:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' pool = image ']'
2024-05-08T01:27:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ case "${pool}" in
2024-05-08T01:27:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:27:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test_force_promote 100
2024-05-08T01:27:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:27:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:27:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:27:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:27:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:27:24.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:27:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:27:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2272.77   bytes/sec: 8.9 MiB/s
2024-05-08T01:27:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_force_promote
2024-05-08T01:27:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test_force_promote stopped
2024-05-08T01:27:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:27:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:27:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test_force_promote stopped
2024-05-08T01:27:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:27:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:27:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:27:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_force_promote
2024-05-08T01:27:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:27:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:27:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:27:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:27:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:27:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:27:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote
2024-05-08T01:27:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:27:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:27:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:27:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:27:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:27:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:27:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:27:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:27:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:27:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:27:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:27:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:27:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:27:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:27:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:27:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote
2024-05-08T01:27:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_force_promote']
2024-05-08T01:27:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_force_promote
2024-05-08T01:27:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:1
2024-05-08T01:27:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:27:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:27:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:27:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:27:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:27:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:27:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:27:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:27:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:27:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:1
2024-05-08T01:27:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:27:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:27:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:1
2024-05-08T01:27:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:27:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster2:1 ']'
2024-05-08T01:27:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:27:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:27:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:27:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:27:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:27:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:27:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:27:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:27:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_force_promote
2024-05-08T01:27:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_force_promote']
2024-05-08T01:27:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:2 rbd mirror status mirror/test_force_promote
2024-05-08T01:27:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:2
2024-05-08T01:27:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:27:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:27:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:27:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:27:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:27:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:27:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:27:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:27:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:27:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:2
2024-05-08T01:27:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:27:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:27:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:2
2024-05-08T01:27:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:27:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster2:2 ']'
2024-05-08T01:27:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:27:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:27:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:27:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:27:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:27:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:27:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:27:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:27:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status mirror/test_force_promote
2024-05-08T01:27:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_force_promote']
2024-05-08T01:27:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:3 rbd mirror status mirror/test_force_promote
2024-05-08T01:27:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:3
2024-05-08T01:27:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:27:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:27:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:27:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:27:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:27:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:27:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:27:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:27:25.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:27:25.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:3
2024-05-08T01:27:25.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:27:25.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:25.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:27:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:3
2024-05-08T01:27:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:27:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster2:3 ']'
2024-05-08T01:27:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:27:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:27:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:27:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:27:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:27:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:27:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:27:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:27:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status mirror/test_force_promote
2024-05-08T01:27:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_force_promote']
2024-05-08T01:27:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 1
2024-05-08T01:27:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:27:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote
2024-05-08T01:27:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:27:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:27:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:27:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:27:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:27:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:27:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:27:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:27:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:27:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:27:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:27:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:27:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:27:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:27:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:27:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote
2024-05-08T01:27:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:27:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:27:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:27:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:27:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:27:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test_force_promote
2024-05-08T01:27:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test_force_promote started
2024-05-08T01:27:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:27:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:27:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test_force_promote started
2024-05-08T01:27:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:27:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:27:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:27:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_force_promote
2024-05-08T01:27:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:27:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:27:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:27:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:27:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:27:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote
2024-05-08T01:27:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:27:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:27:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:27:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote
2024-05-08T01:27:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_force_promote']
2024-05-08T01:27:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote
2024-05-08T01:27:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:27:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:27:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:27:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:27:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:27:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:27:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:27:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:27:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:27:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:27:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:27:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:27:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:27:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:27:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:27:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:27:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:27:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:27:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:27:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote
2024-05-08T01:27:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_force_promote']
2024-05-08T01:27:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_force_promote
2024-05-08T01:27:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:27:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:27:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:27:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:27:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:27:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:27:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:27:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:27:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:27:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:27:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:27:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:27:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:27:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:27:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:27:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:27:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:27:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:27:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:27:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_force_promote
2024-05-08T01:27:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_force_promote']
2024-05-08T01:27:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:28.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:28.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:28.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_force_promote
2024-05-08T01:27:28.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:27:28.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:28.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:28.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:27:28.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:27:28.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:28.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:28.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:28.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:27:28.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:27:28.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:28.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:28.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:27:28.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:27:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:27:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:27:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:27:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:27:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:27:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:27:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:27:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:27:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:27:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:27:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:27:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:27:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_force_promote
2024-05-08T01:27:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_force_promote']
2024-05-08T01:27:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 1
2024-05-08T01:27:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote
2024-05-08T01:27:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:27:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:27:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:27:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote
2024-05-08T01:27:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:27:29.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:27:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:27:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:27:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:27:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:27:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:27:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_force_promote
2024-05-08T01:27:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:27:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:27:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test_force_promote
2024-05-08T01:27:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:27:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:27:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:27:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:27:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:27:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:27:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test_force_promote
2024-05-08T01:27:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:27:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test_force_promote ']'
2024-05-08T01:27:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test_force_promote'
2024-05-08T01:27:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_force_promote
2024-05-08T01:27:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:27:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:27:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:27:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:27:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:27:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_force_promote
2024-05-08T01:27:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:27:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:27:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:27:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:27:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:27:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:27:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:27:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:27:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:27:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:27:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:27:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:27:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:27:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:29.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_force_promote
2024-05-08T01:27:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test_force_promote
2024-05-08T01:27:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_force_promote
2024-05-08T01:27:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_force_promote ''
2024-05-08T01:27:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_force_promote
2024-05-08T01:27:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:27:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_force_promote.status
2024-05-08T01:27:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_force_promote.status
2024-05-08T01:27:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:27:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_force_promote.status
2024-05-08T01:27:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_force_promote
2024-05-08T01:27:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_force_promote.status
2024-05-08T01:27:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:27:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:27:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:27:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:27:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected]
2024-05-08T01:27:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_force_promote.status
2024-05-08T01:27:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:27:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test_force_promote
2024-05-08T01:27:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_force_promote
2024-05-08T01:27:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_force_promote '..*'
2024-05-08T01:27:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_force_promote
2024-05-08T01:27:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:27:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_force_promote.status
2024-05-08T01:27:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_force_promote.status
2024-05-08T01:27:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:27:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_force_promote.status
2024-05-08T01:27:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_force_promote
2024-05-08T01:27:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_force_promote.status
2024-05-08T01:27:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:27:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:27:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:27:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:27:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected]
2024-05-08T01:27:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_force_promote.status
2024-05-08T01:27:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1'
2024-05-08T01:27:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=1, tag_tid=2, entry_tid=1'
2024-05-08T01:27:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' ''
2024-05-08T01:27:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:27:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' = ''
2024-05-08T01:27:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=1, tag_tid=2, entry_tid=1'
2024-05-08T01:27:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:27:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:27:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:27:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=2
2024-05-08T01:27:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1'
2024-05-08T01:27:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:27:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:27:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=2
2024-05-08T01:27:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:27:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:27:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:27:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=3
2024-05-08T01:27:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1'
2024-05-08T01:27:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:27:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:27:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=1
2024-05-08T01:27:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 2 = 2 -a 3 -le 1
2024-05-08T01:27:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:27:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:27:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:27:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test_force_promote
2024-05-08T01:27:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:27:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test_force_promote ']'
2024-05-08T01:27:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test_force_promote'
2024-05-08T01:27:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_force_promote
2024-05-08T01:27:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:27:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:27:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:27:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:27:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:27:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_force_promote
2024-05-08T01:27:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:27:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:27:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:27:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:27:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:27:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:27:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:27:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:27:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:27:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:27:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:27:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:27:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:27:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_force_promote
2024-05-08T01:27:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test_force_promote
2024-05-08T01:27:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_force_promote
2024-05-08T01:27:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_force_promote ''
2024-05-08T01:27:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_force_promote
2024-05-08T01:27:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:27:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_force_promote.status
2024-05-08T01:27:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_force_promote.status
2024-05-08T01:27:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:27:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_force_promote.status
2024-05-08T01:27:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_force_promote
2024-05-08T01:27:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_force_promote.status
2024-05-08T01:27:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:27:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:27:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:27:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:27:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:27:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_force_promote.status
2024-05-08T01:27:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:27:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test_force_promote
2024-05-08T01:27:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_force_promote
2024-05-08T01:27:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_force_promote '..*'
2024-05-08T01:27:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_force_promote
2024-05-08T01:27:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:27:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_force_promote.status
2024-05-08T01:27:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_force_promote.status
2024-05-08T01:27:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:27:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_force_promote.status
2024-05-08T01:27:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_force_promote
2024-05-08T01:27:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_force_promote.status
2024-05-08T01:27:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:27:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:27:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:27:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:27:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:27:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_force_promote.status
2024-05-08T01:27:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:27:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:27:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test_force_promote
2024-05-08T01:27:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:27:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_force_promote up+replaying primary_position
2024-05-08T01:27:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:27:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:27:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test_force_promote up+replaying primary_position ''
2024-05-08T01:27:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:27:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:27:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_force_promote
2024-05-08T01:27:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test_force_promote:
2024-05-08T01:27:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   17b36e7b-dd4a-4023-85f3-04a78ab8b133
2024-05-08T01:27:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:27:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:27:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:27:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:27:29
2024-05-08T01:27:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:27:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:27:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:27:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:27:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:27:27'
2024-05-08T01:27:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:27:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:27:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:27:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:27:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:27:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:27:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:27:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:27:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:27:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:27:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:27:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:27:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:29'
2024-05-08T01:27:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_force_promote up+replaying '2024-05-08 01:27:29'
2024-05-08T01:27:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:27:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:27:29'
2024-05-08T01:27:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:27:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>2</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:115},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:118}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:27:19</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:27:27</last_update></peer_site></peer_sites></image><image><name>test_force_promote</name><global_id>17b36e7b-dd4a-4023-85f3-04a78ab8b133</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:1},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:27:29</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:27:27</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:27:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:27:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update'
2024-05-08T01:27:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:29'
2024-05-08T01:27:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state'
2024-05-08T01:27:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:27:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:27:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:27:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:27:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_force_promote up+stopped
2024-05-08T01:27:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:27:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test_force_promote up+stopped '' ''
2024-05-08T01:27:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:27:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_force_promote
2024-05-08T01:27:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test_force_promote:
2024-05-08T01:27:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   17b36e7b-dd4a-4023-85f3-04a78ab8b133
2024-05-08T01:27:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:27:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:27:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:27:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:27:32
2024-05-08T01:27:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:27:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:27:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+replaying
2024-05-08T01:27:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:27:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:27:29'
2024-05-08T01:27:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:27:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:27:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:27:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:27:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:27:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:27:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:27:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:27:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:27:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:27:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:27:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:27:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:32'
2024-05-08T01:27:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_force_promote up+stopped '2024-05-08 01:27:32'
2024-05-08T01:27:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:27:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:27:32'
2024-05-08T01:27:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:27:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>2</replaying></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:27:32</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:1,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:115},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:118}}</description><last_update>2024-05-08 01:27:19</last_update></peer_site></peer_sites></image><image><name>test_force_promote</name><global_id>17b36e7b-dd4a-4023-85f3-04a78ab8b133</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:27:32</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:1},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><last_update>2024-05-08 01:27:29</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:27:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:27:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update'
2024-05-08T01:27:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:32'
2024-05-08T01:27:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state'
2024-05-08T01:27:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:27:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:27:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:27:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:27:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror test_force_promote --force
2024-05-08T01:27:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=--force
2024-05-08T01:27:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test_force_promote --force
2024-05-08T01:27:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:27:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test_force_promote
2024-05-08T01:27:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test_force_promote stopped
2024-05-08T01:27:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:27:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:27:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test_force_promote stopped
2024-05-08T01:27:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:27:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:27:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:27:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_force_promote
2024-05-08T01:27:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:27:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:27:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:27:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:27:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:27:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote
2024-05-08T01:27:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:27:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:27:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:27:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote
2024-05-08T01:27:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:27:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:27:34.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:27:34.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:27:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:27:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_force_promote
2024-05-08T01:27:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster2 mirror test_force_promote stopped
2024-05-08T01:27:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:27:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:27:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster2 mirror test_force_promote stopped
2024-05-08T01:27:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:27:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:27:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:27:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:27:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_force_promote
2024-05-08T01:27:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster2
2024-05-08T01:27:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster2
2024-05-08T01:27:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:27:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:27:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:27:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote
2024-05-08T01:27:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:27:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:27:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:27:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:27:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:27:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:27:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:27:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster2:0
2024-05-08T01:27:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2
2024-05-08T01:27:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:27:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:27:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:27:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:27:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster2
2024-05-08T01:27:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster2
2024-05-08T01:27:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:27:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:27:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote
2024-05-08T01:27:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:27:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:27:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:27:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:27:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:27:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_force_promote up+stopped
2024-05-08T01:27:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:27:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test_force_promote up+stopped '' ''
2024-05-08T01:27:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:27:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_force_promote
2024-05-08T01:27:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test_force_promote:
2024-05-08T01:27:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   17b36e7b-dd4a-4023-85f3-04a78ab8b133
2024-05-08T01:27:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:27:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:27:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:27:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:27:34
2024-05-08T01:27:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:27:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:27:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:27:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:27:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:27:32'
2024-05-08T01:27:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:27:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:27:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:27:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:27:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:27:36.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:27:36.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:27:36.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:27:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:27:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:27:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:27:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:27:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:34'
2024-05-08T01:27:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_force_promote up+stopped '2024-05-08 01:27:34'
2024-05-08T01:27:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:27:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:27:34'
2024-05-08T01:27:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:27:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying><stopped>1</stopped></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:2.4,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.1,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:118},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:118}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:27:34</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:27:32</last_update></peer_site></peer_sites></image><image><name>test_force_promote</name><global_id>17b36e7b-dd4a-4023-85f3-04a78ab8b133</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:27:34</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:27:32</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:27:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:27:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update'
2024-05-08T01:27:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:34'
2024-05-08T01:27:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state'
2024-05-08T01:27:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:27:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:27:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:27:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:27:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_force_promote up+stopped
2024-05-08T01:27:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:27:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster2 mirror test_force_promote up+stopped '' ''
2024-05-08T01:27:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:27:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:27:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_force_promote
2024-05-08T01:27:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test_force_promote:
2024-05-08T01:27:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   17b36e7b-dd4a-4023-85f3-04a78ab8b133
2024-05-08T01:27:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:27:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: local image is primary
2024-05-08T01:27:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:27:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:27:37
2024-05-08T01:27:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:27:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster1
2024-05-08T01:27:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:27:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:27:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:27:34'
2024-05-08T01:27:37.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:27:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:27:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:27:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: local image is primary
2024-05-08T01:27:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:27:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:27:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:27:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:27:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:27:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:27:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:27:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:27:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:37'
2024-05-08T01:27:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_force_promote up+stopped '2024-05-08 01:27:37'
2024-05-08T01:27:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:27:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:27:37'
2024-05-08T01:27:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml
2024-05-08T01:27:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying><stopped>1</stopped></states></summary><daemons><daemon><service_id>4467</service_id><instance_id>4474</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4471</service_id><instance_id>4481</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4478</service_id><instance_id>4491</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4485</service_id><instance_id>4493</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:27:37</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:2.4,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.1,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:118},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:118}}</description><last_update>2024-05-08 01:27:34</last_update></peer_site></peer_sites></image><image><name>test_force_promote</name><global_id>17b36e7b-dd4a-4023-85f3-04a78ab8b133</global_id><state>up+stopped</state><description>local image is primary</description><daemon_service><service_id>4467</service_id><instance_id>4474</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:27:37</last_update><peer_sites><peer_site><site_name>cluster1</site_name><mirror_uuids>d8559015-eaca-4ba8-a079-91f9b3432357</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:27:34</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:27:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:27:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update'
2024-05-08T01:27:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:37'
2024-05-08T01:27:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state'
2024-05-08T01:27:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:27:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:27:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:27:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:27:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster1 mirror test_force_promote 100
2024-05-08T01:27:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:27:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:27:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:27:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:27:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 bench mirror/test_force_promote --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:27:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:27:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:27:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 1470.61   bytes/sec: 5.7 MiB/s
2024-05-08T01:27:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test_force_promote 100
2024-05-08T01:27:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:27:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:27:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:27:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:27:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:27:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:27:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:27:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2500.04   bytes/sec: 9.8 MiB/s
2024-05-08T01:27:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster1 mirror test_force_promote
2024-05-08T01:27:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:27:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:27:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster1 mirror test_force_promote
2024-05-08T01:27:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_force_promote
2024-05-08T01:27:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 rm mirror/test_force_promote
2024-05-08T01:27:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:27:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster2 mirror test_force_promote
2024-05-08T01:27:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:27:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:27:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster2 mirror test_force_promote
2024-05-08T01:27:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_force_promote
2024-05-08T01:27:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_force_promote
2024-05-08T01:27:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rm mirror/test_force_promote
2024-05-08T01:27:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:27:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: cloned images'
2024-05-08T01:27:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:27:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:27:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:27:38 TEST: cloned images
2024-05-08T01:27:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:27:38 TEST: cloned images
2024-05-08T01:27:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog ' - default'
2024-05-08T01:27:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:27:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:27:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:27:38 - default
2024-05-08T01:27:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:27:38 - default
2024-05-08T01:27:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ parent_image=test_parent
2024-05-08T01:27:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ parent_snap=snap
2024-05-08T01:27:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image_and_enable_mirror cluster2 mirror_parent test_parent journal
2024-05-08T01:27:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:27:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n journal ']'
2024-05-08T01:27:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image cluster2 mirror_parent test_parent
2024-05-08T01:27:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=128
2024-05-08T01:27:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:27:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror_parent/test_parent
2024-05-08T01:27:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ is_pool_mirror_mode_image mirror_parent
2024-05-08T01:27:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' pool = image ']'
2024-05-08T01:27:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ case "${pool}" in
2024-05-08T01:27:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ enable_mirror cluster2 mirror_parent test_parent journal
2024-05-08T01:27:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:27:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image enable mirror_parent/test_parent journal
2024-05-08T01:27:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Mirroring enabled
2024-05-08T01:27:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 info mirror_parent/test_parent
2024-05-08T01:27:39.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:rbd image 'test_parent':
2024-05-08T01:27:39.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	size 128 MiB in 32 objects
2024-05-08T01:27:39.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	order 22 (4 MiB objects)
2024-05-08T01:27:39.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	snapshot_count: 0
2024-05-08T01:27:39.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	id: 15df7b4b0816
2024-05-08T01:27:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	block_name_prefix: rbd_data.15df7b4b0816
2024-05-08T01:27:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	format: 2
2024-05-08T01:27:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	features: layering, exclusive-lock, journaling
2024-05-08T01:27:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	op_features:
2024-05-08T01:27:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	flags:
2024-05-08T01:27:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	create_timestamp: Wed May  8 01:27:38 2024
2024-05-08T01:27:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	access_timestamp: Wed May  8 01:27:38 2024
2024-05-08T01:27:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	modify_timestamp: Wed May  8 01:27:38 2024
2024-05-08T01:27:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	journal: 15df7b4b0816
2024-05-08T01:27:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring state: enabled
2024-05-08T01:27:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring mode: journal
2024-05-08T01:27:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring global id: 90dc459d-a66c-4fb8-8a13-4a66882c7a52
2024-05-08T01:27:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring primary: true
2024-05-08T01:27:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror_parent test_parent 100
2024-05-08T01:27:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:27:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:27:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:27:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:27:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror_parent/test_parent --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:27:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:27:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:27:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2500.04   bytes/sec: 9.8 MiB/s
2024-05-08T01:27:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_snapshot cluster2 mirror_parent test_parent snap
2024-05-08T01:27:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap
2024-05-08T01:27:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap create mirror_parent/test_parent@snap
2024-05-08T01:27:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Creating snap: 10% complete...
Creating snap: 100% complete...done.
2024-05-08T01:27:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ protect_snapshot cluster2 mirror_parent test_parent snap
2024-05-08T01:27:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap
2024-05-08T01:27:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap protect mirror_parent/test_parent@snap
2024-05-08T01:27:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ clone_image=test_clone
2024-05-08T01:27:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ clone_image cluster2 mirror_parent test_parent snap mirror test_clone
2024-05-08T01:27:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_pool=mirror_parent
2024-05-08T01:27:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_image=test_parent
2024-05-08T01:27:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_snap=snap
2024-05-08T01:27:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local clone_pool=mirror
2024-05-08T01:27:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local clone_image=test_clone
2024-05-08T01:27:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift 6
2024-05-08T01:27:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 clone mirror_parent/test_parent@snap mirror/test_clone --image-feature layering,exclusive-lock,journaling
2024-05-08T01:27:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test_clone 100
2024-05-08T01:27:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone
2024-05-08T01:27:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:27:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:27:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:27:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:27:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test_clone --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:27:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:27:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:27:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 1785.74   bytes/sec: 7.0 MiB/s
2024-05-08T01:27:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = snapshot ']'
2024-05-08T01:27:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ enable_mirror cluster2 mirror_parent test_parent journal
2024-05-08T01:27:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:27:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image enable mirror_parent/test_parent journal
2024-05-08T01:27:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Mirroring enabled
2024-05-08T01:27:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 info mirror_parent/test_parent
2024-05-08T01:27:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:rbd image 'test_parent':
2024-05-08T01:27:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	size 128 MiB in 32 objects
2024-05-08T01:27:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	order 22 (4 MiB objects)
2024-05-08T01:27:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	snapshot_count: 1
2024-05-08T01:27:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	id: 15df7b4b0816
2024-05-08T01:27:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	block_name_prefix: rbd_data.15df7b4b0816
2024-05-08T01:27:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	format: 2
2024-05-08T01:27:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	features: layering, exclusive-lock, journaling
2024-05-08T01:27:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	op_features:
2024-05-08T01:27:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	flags:
2024-05-08T01:27:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	create_timestamp: Wed May  8 01:27:38 2024
2024-05-08T01:27:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	access_timestamp: Wed May  8 01:27:38 2024
2024-05-08T01:27:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	modify_timestamp: Wed May  8 01:27:38 2024
2024-05-08T01:27:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	journal: 15df7b4b0816
2024-05-08T01:27:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring state: enabled
2024-05-08T01:27:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring mode: journal
2024-05-08T01:27:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring global id: 90dc459d-a66c-4fb8-8a13-4a66882c7a52
2024-05-08T01:27:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring primary: true
2024-05-08T01:27:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror_parent test_parent
2024-05-08T01:27:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror_parent test_parent started
2024-05-08T01:27:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:27:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:27:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror_parent test_parent started
2024-05-08T01:27:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:27:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:27:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:27:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror_parent/test_parent
2024-05-08T01:27:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:27:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:27:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:27:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:27:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:27:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror_parent/test_parent
2024-05-08T01:27:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:27:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:27:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:27:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror_parent/test_parent
2024-05-08T01:27:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent']
2024-05-08T01:27:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror_parent/test_parent
2024-05-08T01:27:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:27:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:27:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:27:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:27:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:27:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:27:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:27:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:27:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:27:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:27:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:27:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:27:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:27:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:27:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:27:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:27:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:27:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:27:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:27:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror_parent/test_parent
2024-05-08T01:27:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent']
2024-05-08T01:27:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror_parent/test_parent
2024-05-08T01:27:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:27:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:27:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:27:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:27:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:27:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:27:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:27:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:27:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:27:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:27:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:27:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:27:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:27:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:27:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:27:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:27:41.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:27:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:27:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:27:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror_parent/test_parent
2024-05-08T01:27:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent']
2024-05-08T01:27:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror_parent/test_parent
2024-05-08T01:27:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:27:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:27:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:27:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:27:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:27:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:27:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:27:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:27:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:27:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:27:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:27:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:27:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:27:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:27:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:27:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:27:41.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:27:41.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:27:41.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:27:41.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror_parent/test_parent
2024-05-08T01:27:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent']
2024-05-08T01:27:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 1
2024-05-08T01:27:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror_parent/test_parent
2024-05-08T01:27:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:27:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:42.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:42.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:27:42.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:42.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:42.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:42.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:27:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:42.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:42.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:42.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror_parent/test_parent
2024-05-08T01:27:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent']
2024-05-08T01:27:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:42.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:42.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror_parent/test_parent
2024-05-08T01:27:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:27:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:27:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:27:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:27:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:27:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:27:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:27:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:27:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:27:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:27:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:27:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:27:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:27:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:27:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:27:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:27:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:27:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:27:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:27:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror_parent/test_parent
2024-05-08T01:27:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent']
2024-05-08T01:27:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror_parent/test_parent
2024-05-08T01:27:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:27:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:27:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:27:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:27:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:27:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:27:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:27:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:27:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:27:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:27:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:27:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:27:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:27:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:27:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:27:42.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:27:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:27:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:27:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:27:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror_parent/test_parent
2024-05-08T01:27:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent']
2024-05-08T01:27:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror_parent/test_parent
2024-05-08T01:27:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:27:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:27:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:27:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:27:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:27:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:27:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:27:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:27:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:27:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:27:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:27:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:27:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:27:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:27:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:27:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:27:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:27:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:27:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:27:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror_parent/test_parent
2024-05-08T01:27:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent']
2024-05-08T01:27:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 2
2024-05-08T01:27:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror_parent/test_parent
2024-05-08T01:27:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:27:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror_parent/test_parent
2024-05-08T01:27:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:27:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:27:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:27:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:27:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:27:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:27:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:27:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror_parent test_parent
2024-05-08T01:27:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:27:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:27:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror_parent test_parent
2024-05-08T01:27:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:27:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:27:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:27:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:27:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:27:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:27:45.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror_parent test_parent
2024-05-08T01:27:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:27:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test_parent ']'
2024-05-08T01:27:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror_parent/test_parent'
2024-05-08T01:27:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror_parent/test_parent
2024-05-08T01:27:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:27:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:27:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:27:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:27:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:27:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror_parent/test_parent
2024-05-08T01:27:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:27:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:27:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:27:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:27:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:27:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:27:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:27:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:27:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:27:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:27:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:27:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:27:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:27:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror_parent/test_parent
2024-05-08T01:27:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror_parent test_parent
2024-05-08T01:27:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror_parent
2024-05-08T01:27:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_parent
2024-05-08T01:27:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror_parent test_parent ''
2024-05-08T01:27:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror_parent
2024-05-08T01:27:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_parent
2024-05-08T01:27:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:27:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror_parent-test_parent.status
2024-05-08T01:27:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:27:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror_parent-test_parent.status
2024-05-08T01:27:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror_parent-test_parent.status
2024-05-08T01:27:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror_parent/test_parent
2024-05-08T01:27:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror_parent-test_parent.status
2024-05-08T01:27:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:27:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:27:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:27:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected]
2024-05-08T01:27:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=5242c13b-3f2a-4058-805b-0bff89e08325, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected]
2024-05-08T01:27:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror_parent-test_parent.status
2024-05-08T01:27:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=3, tag_tid=4, entry_tid=3'
2024-05-08T01:27:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror_parent test_parent
2024-05-08T01:27:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror_parent
2024-05-08T01:27:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_parent
2024-05-08T01:27:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror_parent test_parent '..*'
2024-05-08T01:27:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror_parent
2024-05-08T01:27:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_parent
2024-05-08T01:27:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:27:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror_parent-test_parent.status
2024-05-08T01:27:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror_parent-test_parent.status
2024-05-08T01:27:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:27:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror_parent-test_parent.status
2024-05-08T01:27:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror_parent/test_parent
2024-05-08T01:27:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror_parent-test_parent.status
2024-05-08T01:27:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:27:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:27:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:27:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected]
2024-05-08T01:27:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=5242c13b-3f2a-4058-805b-0bff89e08325, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected]
2024-05-08T01:27:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror_parent-test_parent.status
2024-05-08T01:27:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=3, tag_tid=4, entry_tid=3'
2024-05-08T01:27:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=3, tag_tid=4, entry_tid=3' -a 'object_number=3, tag_tid=4, entry_tid=3' = 'object_number=3, tag_tid=4, entry_tid=3'
2024-05-08T01:27:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror_parent test_parent
2024-05-08T01:27:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:27:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror_parent test_parent up+replaying primary_position
2024-05-08T01:27:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:27:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:27:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror_parent test_parent up+replaying primary_position ''
2024-05-08T01:27:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:27:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:27:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror_parent/test_parent
2024-05-08T01:27:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test_parent:
2024-05-08T01:27:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   90dc459d-a66c-4fb8-8a13-4a66882c7a52
2024-05-08T01:27:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:27:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}}
2024-05-08T01:27:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:27:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:27:43
2024-05-08T01:27:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:27:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:27:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:27:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:27:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:27:42'
2024-05-08T01:27:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:27:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:27:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:27:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}}
2024-05-08T01:27:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:27:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:27:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:27:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:27:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:27:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:27:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:27:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:27:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:43'
2024-05-08T01:27:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror_parent test_parent up+replaying '2024-05-08 01:27:43'
2024-05-08T01:27:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:27:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:27:43'
2024-05-08T01:27:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror_parent --verbose --format xml
2024-05-08T01:27:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4332</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4334</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4336</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4338</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test_parent</name><global_id>90dc459d-a66c-4fb8-8a13-4a66882c7a52</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:1,&quot;object_number&quot;:1,&quot;tag_tid&quot;:3},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:4}}</description><daemon_service><service_id>4307</service_id><instance_id>4332</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:27:43</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>1a5f4b4e-6fbe-4755-aac3-bb2f9f6deb01</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:27:42</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:27:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:27:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_parent'\'']/last_update'
2024-05-08T01:27:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:43'
2024-05-08T01:27:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_parent'\'']/state'
2024-05-08T01:27:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:27:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:27:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:27:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:27:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror_parent test_parent
2024-05-08T01:27:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:27:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror_parent-test_parent.export
2024-05-08T01:27:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror_parent-test_parent.export
2024-05-08T01:27:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:27:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror_parent-test_parent.export
2024-05-08T01:27:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror_parent-test_parent.export
2024-05-08T01:27:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror_parent-test_parent.export
2024-05-08T01:27:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:27:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror_parent-test_parent.export
2024-05-08T01:27:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror_parent-test_parent.export /tmp/tmp.3xddryFtVF/cluster1-mirror_parent-test_parent.export
2024-05-08T01:27:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror_parent/test_parent /tmp/tmp.3xddryFtVF/cluster2-mirror_parent-test_parent.export
2024-05-08T01:27:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:27:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror_parent/test_parent /tmp/tmp.3xddryFtVF/cluster1-mirror_parent-test_parent.export
2024-05-08T01:27:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:27:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror_parent-test_parent.export /tmp/tmp.3xddryFtVF/cluster1-mirror_parent-test_parent.export
2024-05-08T01:27:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror_parent-test_parent.export /tmp/tmp.3xddryFtVF/cluster1-mirror_parent-test_parent.export
2024-05-08T01:27:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test_clone
2024-05-08T01:27:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone
2024-05-08T01:27:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test_clone started
2024-05-08T01:27:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone
2024-05-08T01:27:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:27:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:27:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test_clone started
2024-05-08T01:27:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone
2024-05-08T01:27:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:27:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:27:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:27:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_clone
2024-05-08T01:27:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:27:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:27:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:27:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:27:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:27:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone
2024-05-08T01:27:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:27:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:27:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:27:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone
2024-05-08T01:27:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:27:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:27:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:27:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:27:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:27:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:27:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:27:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_clone
2024-05-08T01:27:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:27:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone
2024-05-08T01:27:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:27:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test_clone
2024-05-08T01:27:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:27:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone
2024-05-08T01:27:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:27:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:27:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:27:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:27:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:27:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test_clone
2024-05-08T01:27:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone
2024-05-08T01:27:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:27:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test_clone ']'
2024-05-08T01:27:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test_clone'
2024-05-08T01:27:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_clone
2024-05-08T01:27:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:27:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:27:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:27:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:27:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:27:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_clone
2024-05-08T01:27:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:27:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:27:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:27:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:27:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:27:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:27:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:27:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:27:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:27:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:27:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:27:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:27:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:27:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_clone
2024-05-08T01:27:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test_clone
2024-05-08T01:27:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_clone
2024-05-08T01:27:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_clone ''
2024-05-08T01:27:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_clone
2024-05-08T01:27:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:27:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_clone.status
2024-05-08T01:27:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_clone.status
2024-05-08T01:27:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:27:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_clone.status
2024-05-08T01:27:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_clone
2024-05-08T01:27:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_clone.status
2024-05-08T01:27:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:27:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:27:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:27:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:27:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:27:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_clone.status
2024-05-08T01:27:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:27:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test_clone
2024-05-08T01:27:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_clone
2024-05-08T01:27:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_clone '..*'
2024-05-08T01:27:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_clone
2024-05-08T01:27:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:27:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_clone.status
2024-05-08T01:27:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_clone.status
2024-05-08T01:27:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:27:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_clone.status
2024-05-08T01:27:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_clone
2024-05-08T01:27:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_clone.status
2024-05-08T01:27:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:27:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:27:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:27:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:27:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:27:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_clone.status
2024-05-08T01:27:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:27:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:27:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test_clone
2024-05-08T01:27:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone
2024-05-08T01:27:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:27:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_clone up+replaying primary_position
2024-05-08T01:27:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone
2024-05-08T01:27:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:27:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:27:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test_clone up+replaying primary_position ''
2024-05-08T01:27:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone
2024-05-08T01:27:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:27:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:27:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:27:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_clone
2024-05-08T01:27:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test_clone:
2024-05-08T01:27:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   30adb6eb-2e40-4be3-a9f6-df680cd4c52e
2024-05-08T01:27:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:27:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:27:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:27:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:27:45
2024-05-08T01:27:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:27:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:27:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:27:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:27:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:27:47'
2024-05-08T01:27:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:27:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:27:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:27:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:27:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:27:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:27:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:27:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:27:50.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:27:50.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:27:50.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:27:50.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:27:50.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:45'
2024-05-08T01:27:50.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_clone up+replaying '2024-05-08 01:27:45'
2024-05-08T01:27:50.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone
2024-05-08T01:27:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:27:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:27:45'
2024-05-08T01:27:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:27:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:27:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:27:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>2</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:2.4,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.1,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:118},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:118}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:27:34</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:27:47</last_update></peer_site></peer_sites></image><image><name>test_clone</name><global_id>30adb6eb-2e40-4be3-a9f6-df680cd4c52e</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:1},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:27:45</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:27:47</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:27:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:27:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_clone'\'']/last_update'
2024-05-08T01:27:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:27:45'
2024-05-08T01:27:50.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_clone'\'']/state'
2024-05-08T01:27:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:27:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:27:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:27:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:27:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror test_clone
2024-05-08T01:27:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone
2024-05-08T01:27:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:27:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror-test_clone.export
2024-05-08T01:27:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:27:50.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror-test_clone.export
2024-05-08T01:27:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_clone.export
2024-05-08T01:27:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror-test_clone.export
2024-05-08T01:27:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror-test_clone.export
2024-05-08T01:27:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:27:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror-test_clone.export
2024-05-08T01:27:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test_clone.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test_clone.export
2024-05-08T01:27:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/test_clone /tmp/tmp.3xddryFtVF/cluster2-mirror-test_clone.export
2024-05-08T01:27:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:27:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/test_clone /tmp/tmp.3xddryFtVF/cluster1-mirror-test_clone.export
2024-05-08T01:27:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:27:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror-test_clone.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test_clone.export
2024-05-08T01:27:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test_clone.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test_clone.export
2024-05-08T01:27:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster2 mirror test_clone
2024-05-08T01:27:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone
2024-05-08T01:27:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:27:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:27:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster2 mirror test_clone
2024-05-08T01:27:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone
2024-05-08T01:27:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_clone
2024-05-08T01:27:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rm mirror/test_clone
2024-05-08T01:27:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:27:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog ' - clone v1'
2024-05-08T01:27:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:27:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:27:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:27:50 - clone v1
2024-05-08T01:27:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:27:50 - clone v1
2024-05-08T01:27:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ clone_image_and_enable_mirror cluster1 mirror_parent test_parent snap mirror test_clone1 journal
2024-05-08T01:27:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_pool=mirror_parent
2024-05-08T01:27:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_image=test_parent
2024-05-08T01:27:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_snap=snap
2024-05-08T01:27:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local clone_pool=mirror
2024-05-08T01:27:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local clone_image=test_clone1
2024-05-08T01:27:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift 6
2024-05-08T01:27:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:27:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n journal ']'
2024-05-08T01:27:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ clone_image cluster1 mirror_parent test_parent snap mirror test_clone1
2024-05-08T01:27:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_pool=mirror_parent
2024-05-08T01:27:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_image=test_parent
2024-05-08T01:27:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_snap=snap
2024-05-08T01:27:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local clone_pool=mirror
2024-05-08T01:27:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local clone_image=test_clone1
2024-05-08T01:27:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift 6
2024-05-08T01:27:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 clone mirror_parent/test_parent@snap mirror/test_clone1 --image-feature layering,exclusive-lock,journaling
2024-05-08T01:27:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ is_pool_mirror_mode_image mirror
2024-05-08T01:27:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' pool = image ']'
2024-05-08T01:27:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ case "${pool}" in
2024-05-08T01:27:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:27:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ clone_image_and_enable_mirror cluster2 mirror_parent test_parent snap mirror test_clone_v1 journal --rbd-default-clone-format 1
2024-05-08T01:27:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_pool=mirror_parent
2024-05-08T01:27:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_image=test_parent
2024-05-08T01:27:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_snap=snap
2024-05-08T01:27:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local clone_pool=mirror
2024-05-08T01:27:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local clone_image=test_clone_v1
2024-05-08T01:27:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift 6
2024-05-08T01:27:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:27:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n journal ']'
2024-05-08T01:27:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ clone_image cluster2 mirror_parent test_parent snap mirror test_clone_v1 --rbd-default-clone-format 1
2024-05-08T01:27:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_pool=mirror_parent
2024-05-08T01:27:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_image=test_parent
2024-05-08T01:27:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_snap=snap
2024-05-08T01:27:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local clone_pool=mirror
2024-05-08T01:27:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local clone_image=test_clone_v1
2024-05-08T01:27:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift 6
2024-05-08T01:27:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 clone mirror_parent/test_parent@snap mirror/test_clone_v1 --image-feature layering,exclusive-lock,journaling --rbd-default-clone-format 1
2024-05-08T01:27:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ is_pool_mirror_mode_image mirror
2024-05-08T01:27:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' pool = image ']'
2024-05-08T01:27:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ case "${pool}" in
2024-05-08T01:27:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:27:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_clone_format cluster2 mirror test_clone_v1
2024-05-08T01:27:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_clone_v1
2024-05-08T01:27:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 info mirror/test_clone_v1
2024-05-08T01:27:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ awk 'BEGIN {
2024-05-08T01:27:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               format = 1
2024-05-08T01:27:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             }
2024-05-08T01:27:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             $1 == "parent:" {
2024-05-08T01:27:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               parent = $2
2024-05-08T01:27:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             }
2024-05-08T01:27:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             /op_features: .*clone-child/ {
2024-05-08T01:27:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               format = 2
2024-05-08T01:27:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             }
2024-05-08T01:27:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             END {
2024-05-08T01:27:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               if (!parent) exit 1
2024-05-08T01:27:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               print format
2024-05-08T01:27:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             }'
2024-05-08T01:27:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 1 = 1
2024-05-08T01:27:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test_clone_v1
2024-05-08T01:27:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone_v1
2024-05-08T01:27:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test_clone_v1 started
2024-05-08T01:27:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone_v1
2024-05-08T01:27:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:27:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:27:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test_clone_v1 started
2024-05-08T01:27:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone_v1
2024-05-08T01:27:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:27:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:27:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:27:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_clone_v1
2024-05-08T01:27:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:27:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:27:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:27:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:27:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:27:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v1
2024-05-08T01:27:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:27:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:27:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:27:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:52.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:52.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:52.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:52.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:52.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:52.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:52.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:52.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:52.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v1
2024-05-08T01:27:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1']
2024-05-08T01:27:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v1
2024-05-08T01:27:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:27:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:27:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:27:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:27:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:27:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:27:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:27:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:27:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:27:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:27:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:27:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:27:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:27:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:27:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:27:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:27:52.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:27:52.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:27:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:27:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v1
2024-05-08T01:27:52.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1']
2024-05-08T01:27:52.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_clone_v1
2024-05-08T01:27:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:27:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:27:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:27:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:27:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:27:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:27:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:27:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:27:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:27:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:27:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:27:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:27:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:27:52.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:52.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:52.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:27:52.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:27:52.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:27:52.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:27:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:27:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:27:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone_v1
2024-05-08T01:27:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1']
2024-05-08T01:27:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_clone_v1
2024-05-08T01:27:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:27:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:27:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:27:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:27:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:27:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:27:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:27:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:27:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:27:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:27:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:27:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:27:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:27:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:27:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:27:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:27:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:27:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:27:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:27:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone_v1
2024-05-08T01:27:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1']
2024-05-08T01:27:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 1
2024-05-08T01:27:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v1
2024-05-08T01:27:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:27:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:27:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:27:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v1
2024-05-08T01:27:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1']
2024-05-08T01:27:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v1
2024-05-08T01:27:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:27:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:27:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:27:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:27:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:27:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:27:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:27:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:27:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:27:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:27:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:27:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:27:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:27:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:27:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:27:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:27:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:27:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:27:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:27:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v1
2024-05-08T01:27:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1']
2024-05-08T01:27:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_clone_v1
2024-05-08T01:27:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:27:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:27:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:27:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:27:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:27:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:27:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:27:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:27:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:27:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:27:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:27:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:27:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:27:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:27:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:27:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:27:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:27:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:27:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:27:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone_v1
2024-05-08T01:27:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1']
2024-05-08T01:27:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_clone_v1
2024-05-08T01:27:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:27:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:27:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:27:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:27:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:27:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:27:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:27:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:27:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:27:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:27:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:27:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:27:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:27:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:27:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:27:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:27:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:27:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:27:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:27:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone_v1
2024-05-08T01:27:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1']
2024-05-08T01:27:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 2
2024-05-08T01:27:56.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:56.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v1
2024-05-08T01:27:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:27:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:27:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:27:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v1
2024-05-08T01:27:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:27:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:27:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:27:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:27:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:27:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:27:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:27:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_clone_format cluster1 mirror test_clone_v1
2024-05-08T01:27:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_clone_v1
2024-05-08T01:27:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 info mirror/test_clone_v1
2024-05-08T01:27:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ awk 'BEGIN {
2024-05-08T01:27:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               format = 1
2024-05-08T01:27:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             }
2024-05-08T01:27:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             $1 == "parent:" {
2024-05-08T01:27:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               parent = $2
2024-05-08T01:27:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             }
2024-05-08T01:27:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             /op_features: .*clone-child/ {
2024-05-08T01:27:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               format = 2
2024-05-08T01:27:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             }
2024-05-08T01:27:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             END {
2024-05-08T01:27:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               if (!parent) exit 1
2024-05-08T01:27:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               print format
2024-05-08T01:27:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             }'
2024-05-08T01:27:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 1 = 1
2024-05-08T01:27:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster2 mirror test_clone_v1
2024-05-08T01:27:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone_v1
2024-05-08T01:27:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:27:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:27:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster2 mirror test_clone_v1
2024-05-08T01:27:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone_v1
2024-05-08T01:27:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_clone_v1
2024-05-08T01:27:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rm mirror/test_clone_v1
2024-05-08T01:27:56.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:27:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster1 mirror test_clone1
2024-05-08T01:27:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone1
2024-05-08T01:27:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:27:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:27:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster1 mirror test_clone1
2024-05-08T01:27:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone1
2024-05-08T01:27:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_clone1
2024-05-08T01:27:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 rm mirror/test_clone1
2024-05-08T01:27:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:27:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ unprotect_snapshot_retry cluster2 mirror_parent test_parent snap
2024-05-08T01:27:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap
2024-05-08T01:27:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:27:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:27:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ unprotect_snapshot cluster2 mirror_parent test_parent snap
2024-05-08T01:27:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap
2024-05-08T01:27:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap unprotect mirror_parent/test_parent@snap
2024-05-08T01:27:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:27:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_snapshot cluster2 mirror_parent test_parent snap
2024-05-08T01:27:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap
2024-05-08T01:27:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rm mirror_parent/test_parent@snap
2024-05-08T01:27:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing snap: 100% complete...done.
2024-05-08T01:27:57.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog ' - clone v2'
2024-05-08T01:27:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:27:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:27:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:27:57 - clone v2
2024-05-08T01:27:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:27:57 - clone v2
2024-05-08T01:27:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ parent_snap=snap_v2
2024-05-08T01:27:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_snapshot cluster2 mirror_parent test_parent snap_v2
2024-05-08T01:27:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:27:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:27:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_v2
2024-05-08T01:27:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap create mirror_parent/test_parent@snap_v2
2024-05-08T01:27:58.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Creating snap: 10% complete...
Creating snap: 100% complete...done.
2024-05-08T01:27:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = snapshot ']'
2024-05-08T01:27:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ clone_image_and_enable_mirror cluster2 mirror_parent test_parent snap_v2 mirror test_clone_v2 journal --rbd-default-clone-format 2
2024-05-08T01:27:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_pool=mirror_parent
2024-05-08T01:27:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_image=test_parent
2024-05-08T01:27:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_snap=snap_v2
2024-05-08T01:27:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local clone_pool=mirror
2024-05-08T01:27:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local clone_image=test_clone_v2
2024-05-08T01:27:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift 6
2024-05-08T01:27:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:27:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n journal ']'
2024-05-08T01:27:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:27:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ clone_image cluster2 mirror_parent test_parent snap_v2 mirror test_clone_v2 --rbd-default-clone-format 2
2024-05-08T01:27:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:27:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_pool=mirror_parent
2024-05-08T01:27:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_image=test_parent
2024-05-08T01:27:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_snap=snap_v2
2024-05-08T01:27:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local clone_pool=mirror
2024-05-08T01:27:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local clone_image=test_clone_v2
2024-05-08T01:27:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift 6
2024-05-08T01:27:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 clone mirror_parent/test_parent@snap_v2 mirror/test_clone_v2 --image-feature layering,exclusive-lock,journaling --rbd-default-clone-format 2
2024-05-08T01:27:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ is_pool_mirror_mode_image mirror
2024-05-08T01:27:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' pool = image ']'
2024-05-08T01:27:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ case "${pool}" in
2024-05-08T01:27:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:27:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_clone_format cluster2 mirror test_clone_v2
2024-05-08T01:27:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:27:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:27:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_clone_v2
2024-05-08T01:27:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 info mirror/test_clone_v2
2024-05-08T01:27:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ awk 'BEGIN {
2024-05-08T01:27:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               format = 1
2024-05-08T01:27:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             }
2024-05-08T01:27:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             $1 == "parent:" {
2024-05-08T01:27:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               parent = $2
2024-05-08T01:27:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             }
2024-05-08T01:27:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             /op_features: .*clone-child/ {
2024-05-08T01:27:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               format = 2
2024-05-08T01:27:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             }
2024-05-08T01:27:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             END {
2024-05-08T01:27:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               if (!parent) exit 1
2024-05-08T01:27:58.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               print format
2024-05-08T01:27:58.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             }'
2024-05-08T01:27:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 2 = 2
2024-05-08T01:27:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test_clone_v2
2024-05-08T01:27:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone_v2
2024-05-08T01:27:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test_clone_v2 started
2024-05-08T01:27:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone_v2
2024-05-08T01:27:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:27:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:27:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:27:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test_clone_v2 started
2024-05-08T01:27:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:27:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:27:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone_v2
2024-05-08T01:27:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:27:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:27:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:27:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_clone_v2
2024-05-08T01:27:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:27:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:27:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:27:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:27:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:27:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:27:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:27:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:27:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:27:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2
2024-05-08T01:27:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:27:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:27:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:27:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:27:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:27:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:27:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:27:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:27:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:27:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:27:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:27:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:27:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:27:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:27:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:27:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2
2024-05-08T01:27:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2']
2024-05-08T01:27:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:27:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:27:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:27:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2
2024-05-08T01:27:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:27:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:27:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:27:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:27:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:27:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:27:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:27:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:27:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:27:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:27:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:27:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:27:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:27:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:27:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:27:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:27:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:27:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:27:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:27:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:27:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:27:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:27:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:27:59.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:27:59.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:27:59.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:27:59.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:27:59.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:27:59.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:27:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:27:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:27:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:27:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2']
2024-05-08T01:28:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:28:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:28:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:28:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:28:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:28:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:28:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:28:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:28:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:28:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:28:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:28:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:28:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:28:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:28:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:28:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:28:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:28:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:28:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:28:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:28:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:28:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2']
2024-05-08T01:28:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:28:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:28:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:28:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:28:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:28:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:28:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:28:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:28:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:28:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:28:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:28:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:28:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:28:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:28:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:28:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:28:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:28:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:28:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:28:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:28:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:28:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2']
2024-05-08T01:28:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:28:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:28:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:28:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 1
2024-05-08T01:28:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:28:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:28:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:28:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:28:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:28:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:28:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:28:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:28:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:28:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:28:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:28:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:28:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2']
2024-05-08T01:28:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:28:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:28:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:28:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:28:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:28:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:28:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:28:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:28:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:28:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:28:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:28:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:28:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:28:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:28:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:28:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:28:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:28:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:28:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:28:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:28:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:28:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2']
2024-05-08T01:28:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:28:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:28:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:28:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:28:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:28:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:28:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:28:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:28:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:28:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:28:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:28:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:28:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:28:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:28:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:28:01.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:01.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:01.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:28:01.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:28:01.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:28:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:28:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:28:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:28:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2']
2024-05-08T01:28:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:28:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:28:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:28:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:28:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:28:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:28:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:28:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:28:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:28:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:28:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:28:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:28:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:28:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:28:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:28:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:28:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:28:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:28:01.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:28:01.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:28:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:28:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2']
2024-05-08T01:28:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:28:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:28:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:28:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 2
2024-05-08T01:28:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:28:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:28:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:28:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:28:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:28:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:28:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:28:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:28:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:28:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:28:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:28:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:28:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:28:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:28:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:28:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:28:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2']
2024-05-08T01:28:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:28:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:28:04.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:04.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:04.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:28:04.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:04.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:04.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:28:04.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:28:04.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:04.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:04.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:04.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:28:04.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:28:04.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:04.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:04.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:28:04.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:28:04.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:28:04.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:28:04.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:04.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:28:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:28:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:28:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:28:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:28:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:28:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:28:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:28:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:28:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:28:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2']
2024-05-08T01:28:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:28:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:28:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:28:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:28:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:28:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:28:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:28:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:28:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:28:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:28:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:28:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:28:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:28:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:28:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:28:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:28:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:28:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:28:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:28:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:28:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:28:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2']
2024-05-08T01:28:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:28:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:28:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:28:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:28:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:28:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:28:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:28:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:28:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:28:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:28:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:28:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:28:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:28:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:28:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:28:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:28:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:28:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:28:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:28:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:28:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:28:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2']
2024-05-08T01:28:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:28:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:28:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:28:04.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 4
2024-05-08T01:28:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:28:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:28:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:28:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:28:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:28:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:28:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:28:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:28:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:28:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:28:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:28:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:28:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:28:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:28:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:28:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:28:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2
2024-05-08T01:28:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:28:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:28:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:28:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:28:08.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:28:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:28:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:28:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_clone_format cluster1 mirror test_clone_v2
2024-05-08T01:28:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_clone_v2
2024-05-08T01:28:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 info mirror/test_clone_v2
2024-05-08T01:28:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ awk 'BEGIN {
2024-05-08T01:28:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               format = 1
2024-05-08T01:28:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             }
2024-05-08T01:28:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             $1 == "parent:" {
2024-05-08T01:28:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               parent = $2
2024-05-08T01:28:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             }
2024-05-08T01:28:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             /op_features: .*clone-child/ {
2024-05-08T01:28:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               format = 2
2024-05-08T01:28:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             }
2024-05-08T01:28:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             END {
2024-05-08T01:28:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               if (!parent) exit 1
2024-05-08T01:28:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:               print format
2024-05-08T01:28:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:             }'
2024-05-08T01:28:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 2 = 2
2024-05-08T01:28:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_snapshot cluster2 mirror_parent test_parent snap_v2
2024-05-08T01:28:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:28:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:28:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_v2
2024-05-08T01:28:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rm mirror_parent/test_parent@snap_v2
2024-05-08T01:28:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing snap: 100% complete...done.
2024-05-08T01:28:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = snapshot ']'
2024-05-08T01:28:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_snap_moved_to_trash cluster2 mirror_parent test_parent snap_v2
2024-05-08T01:28:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:28:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:28:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap_name=snap_v2
2024-05-08T01:28:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap ls mirror_parent/test_parent --all
2024-05-08T01:28:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -F ' trash (snap_v2)'
2024-05-08T01:28:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:     5  69582d39-5045-4053-8e68-ff1bec4d78ed  128 MiB             Wed May  8 01:27:58 2024  trash (snap_v2)
2024-05-08T01:28:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2
2024-05-08T01:28:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:28:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:28:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:28:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2
2024-05-08T01:28:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:28:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:28:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap_name=snap_v2
2024-05-08T01:28:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all
2024-05-08T01:28:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -F ' trash (snap_v2)'
2024-05-08T01:28:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ continue
2024-05-08T01:28:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:28:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:28:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2
2024-05-08T01:28:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:28:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:28:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap_name=snap_v2
2024-05-08T01:28:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all
2024-05-08T01:28:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -F ' trash (snap_v2)'
2024-05-08T01:28:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:     5  9b843ab2-d1d4-400f-b69c-de67c79b7735  128 MiB             Wed May  8 01:28:01 2024  trash (snap_v2)
2024-05-08T01:28:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster2 mirror test_clone_v2
2024-05-08T01:28:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone_v2
2024-05-08T01:28:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:28:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:28:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster2 mirror test_clone_v2
2024-05-08T01:28:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone_v2
2024-05-08T01:28:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_clone_v2
2024-05-08T01:28:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rm mirror/test_clone_v2
2024-05-08T01:28:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:28:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test_clone_v2 deleted
2024-05-08T01:28:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone_v2
2024-05-08T01:28:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=deleted
2024-05-08T01:28:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:28:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:28:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:28:12.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test_clone_v2
2024-05-08T01:28:12.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:12.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_clone_v2
2024-05-08T01:28:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test_clone_v2
2024-05-08T01:28:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:28:12.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test_clone_v2: (2) No such file or directory
2024-05-08T01:28:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=
2024-05-08T01:28:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:28:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:28:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test_clone_v2 deleted ''
2024-05-08T01:28:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone_v2
2024-05-08T01:28:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:28:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:28:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:28:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:28:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test_clone_v2
2024-05-08T01:28:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_clone_v2
2024-05-08T01:28:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test_clone_v2
2024-05-08T01:28:12.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:28:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test_clone_v2: (2) No such file or directory
2024-05-08T01:28:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:28:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:28:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = deleted
2024-05-08T01:28:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_snap_removed_from_trash cluster2 mirror_parent test_parent snap_v2
2024-05-08T01:28:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_snap_moved_to_trash cluster2 mirror_parent test_parent snap_v2
2024-05-08T01:28:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:28:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:28:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap_name=snap_v2
2024-05-08T01:28:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap ls mirror_parent/test_parent --all
2024-05-08T01:28:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -F ' trash (snap_v2)'
2024-05-08T01:28:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2
2024-05-08T01:28:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:28:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:28:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:28:13.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2
2024-05-08T01:28:13.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2
2024-05-08T01:28:13.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:13.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:28:13.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:28:13.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap_name=snap_v2
2024-05-08T01:28:13.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all
2024-05-08T01:28:13.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -F ' trash (snap_v2)'
2024-05-08T01:28:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog ' - clone v2 non-primary'
2024-05-08T01:28:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:28:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:28:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:28:14 - clone v2 non-primary
2024-05-08T01:28:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:28:14 - clone v2 non-primary
2024-05-08T01:28:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_snapshot cluster2 mirror_parent test_parent snap_v2
2024-05-08T01:28:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:28:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:28:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_v2
2024-05-08T01:28:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap create mirror_parent/test_parent@snap_v2
2024-05-08T01:28:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Creating snap: 10% complete...
Creating snap: 100% complete...done.
2024-05-08T01:28:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = snapshot ']'
2024-05-08T01:28:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_snap_present cluster1 mirror_parent test_parent snap_v2
2024-05-08T01:28:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:28:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:28:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap_name=snap_v2
2024-05-08T01:28:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:28:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:28:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:28:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 info mirror_parent/test_parent@snap_v2
2024-05-08T01:28:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:error setting snapshot context: (2) No such file or directory
2024-05-08T01:28:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ continue
2024-05-08T01:28:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:28:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:28:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 info mirror_parent/test_parent@snap_v2
2024-05-08T01:28:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:rbd image 'test_parent':
2024-05-08T01:28:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	size 128 MiB in 32 objects
2024-05-08T01:28:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	order 22 (4 MiB objects)
2024-05-08T01:28:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	snapshot_count: 1
2024-05-08T01:28:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	id: 10eceacca181
2024-05-08T01:28:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	block_name_prefix: rbd_data.10eceacca181
2024-05-08T01:28:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	format: 2
2024-05-08T01:28:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	features: layering, exclusive-lock, journaling
2024-05-08T01:28:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	op_features:
2024-05-08T01:28:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	flags:
2024-05-08T01:28:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	create_timestamp: Wed May  8 01:27:41 2024
2024-05-08T01:28:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	access_timestamp: Wed May  8 01:27:41 2024
2024-05-08T01:28:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	modify_timestamp: Wed May  8 01:27:41 2024
2024-05-08T01:28:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	protected: False
2024-05-08T01:28:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	journal: 10eceacca181
2024-05-08T01:28:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring state: enabled
2024-05-08T01:28:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring mode: journal
2024-05-08T01:28:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring global id: 90dc459d-a66c-4fb8-8a13-4a66882c7a52
2024-05-08T01:28:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring primary: false
2024-05-08T01:28:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ clone_image_and_enable_mirror cluster1 mirror_parent test_parent snap_v2 mirror test_clone_v2 journal --rbd-default-clone-format 2
2024-05-08T01:28:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_pool=mirror_parent
2024-05-08T01:28:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_image=test_parent
2024-05-08T01:28:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_snap=snap_v2
2024-05-08T01:28:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local clone_pool=mirror
2024-05-08T01:28:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local clone_image=test_clone_v2
2024-05-08T01:28:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift 6
2024-05-08T01:28:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:28:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n journal ']'
2024-05-08T01:28:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ clone_image cluster1 mirror_parent test_parent snap_v2 mirror test_clone_v2 --rbd-default-clone-format 2
2024-05-08T01:28:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_pool=mirror_parent
2024-05-08T01:28:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_image=test_parent
2024-05-08T01:28:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local parent_snap=snap_v2
2024-05-08T01:28:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local clone_pool=mirror
2024-05-08T01:28:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local clone_image=test_clone_v2
2024-05-08T01:28:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift 6
2024-05-08T01:28:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 clone mirror_parent/test_parent@snap_v2 mirror/test_clone_v2 --image-feature layering,exclusive-lock,journaling --rbd-default-clone-format 2
2024-05-08T01:28:17.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ is_pool_mirror_mode_image mirror
2024-05-08T01:28:17.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' pool = image ']'
2024-05-08T01:28:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ case "${pool}" in
2024-05-08T01:28:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:28:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_snapshot cluster2 mirror_parent test_parent snap_v2
2024-05-08T01:28:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:28:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:28:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_v2
2024-05-08T01:28:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rm mirror_parent/test_parent@snap_v2
2024-05-08T01:28:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing snap: 100% complete...done.
2024-05-08T01:28:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_snap_removed_from_trash cluster2 mirror_parent test_parent snap_v2
2024-05-08T01:28:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_snap_moved_to_trash cluster2 mirror_parent test_parent snap_v2
2024-05-08T01:28:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:28:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:28:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap_name=snap_v2
2024-05-08T01:28:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap ls mirror_parent/test_parent --all
2024-05-08T01:28:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -F ' trash (snap_v2)'
2024-05-08T01:28:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = snapshot ']'
2024-05-08T01:28:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2
2024-05-08T01:28:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:28:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:28:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:28:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2
2024-05-08T01:28:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:28:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:28:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap_name=snap_v2
2024-05-08T01:28:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all
2024-05-08T01:28:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -F ' trash (snap_v2)'
2024-05-08T01:28:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:     6  77a374b5-bcab-42f9-b05b-d66132d8dfc4  128 MiB             Wed May  8 01:28:16 2024  trash (snap_v2)
2024-05-08T01:28:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster1 mirror test_clone_v2
2024-05-08T01:28:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone_v2
2024-05-08T01:28:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:28:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:28:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster1 mirror test_clone_v2
2024-05-08T01:28:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_clone_v2
2024-05-08T01:28:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_clone_v2
2024-05-08T01:28:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 rm mirror/test_clone_v2
2024-05-08T01:28:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:28:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2
2024-05-08T01:28:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:28:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:28:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:28:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2
2024-05-08T01:28:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2
2024-05-08T01:28:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:28:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:28:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap_name=snap_v2
2024-05-08T01:28:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all
2024-05-08T01:28:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -F ' trash (snap_v2)'
2024-05-08T01:28:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster2 mirror_parent test_parent
2024-05-08T01:28:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:28:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:28:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:28:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:28:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster2 mirror_parent test_parent
2024-05-08T01:28:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror_parent
2024-05-08T01:28:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_parent
2024-05-08T01:28:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 snap purge mirror_parent/test_parent
2024-05-08T01:28:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rm mirror_parent/test_parent
2024-05-08T01:28:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:28:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: data pool'
2024-05-08T01:28:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:28:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:28:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:28:21 TEST: data pool
2024-05-08T01:28:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:28:21 TEST: data pool
2024-05-08T01:28:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ dp_image=test_data_pool
2024-05-08T01:28:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image_and_enable_mirror cluster2 mirror test_data_pool journal 128 --data-pool mirror_parent
2024-05-08T01:28:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:28:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n journal ']'
2024-05-08T01:28:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image cluster2 mirror test_data_pool 128 --data-pool mirror_parent
2024-05-08T01:28:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=128
2024-05-08T01:28:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 128 ']'
2024-05-08T01:28:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=128
2024-05-08T01:28:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --data-pool mirror_parent mirror/test_data_pool
2024-05-08T01:28:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ is_pool_mirror_mode_image mirror
2024-05-08T01:28:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' pool = image ']'
2024-05-08T01:28:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ case "${pool}" in
2024-05-08T01:28:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:28:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_data_pool cluster2 mirror test_data_pool
2024-05-08T01:28:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 info mirror/test_data_pool
2024-05-08T01:28:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ awk '$1 == "data_pool:" {print $2}'
2024-05-08T01:28:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ data_pool=mirror_parent
2024-05-08T01:28:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test mirror_parent = mirror_parent
2024-05-08T01:28:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test_data_pool
2024-05-08T01:28:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test_data_pool started
2024-05-08T01:28:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:28:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:28:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:28:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:28:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test_data_pool started
2024-05-08T01:28:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:28:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:28:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:28:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_data_pool
2024-05-08T01:28:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:28:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:28:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:28:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:28:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:28:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:28:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:28:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:28:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_data_pool
2024-05-08T01:28:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:28:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:28:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:28:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:28:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:28:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:28:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:28:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:28:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:28:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:28:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:28:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:28:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:28:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:28:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_data_pool
2024-05-08T01:28:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_data_pool']
2024-05-08T01:28:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:28:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:28:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_data_pool
2024-05-08T01:28:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:28:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:28:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:28:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:28:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:28:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:28:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:28:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:28:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:28:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:28:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:28:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:28:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:28:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:28:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:28:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:28:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:28:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:28:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:28:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_data_pool
2024-05-08T01:28:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_data_pool']
2024-05-08T01:28:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:28:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:28:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_data_pool
2024-05-08T01:28:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:28:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:28:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:28:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:28:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:28:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:28:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:28:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:28:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:28:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:28:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:28:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:28:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:28:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:28:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:28:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:28:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:28:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:28:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:28:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_data_pool
2024-05-08T01:28:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_data_pool']
2024-05-08T01:28:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:28:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:28:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_data_pool
2024-05-08T01:28:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:28:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:28:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:28:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:28:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:28:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:28:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:28:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:28:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:28:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:28:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:28:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:28:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:28:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:28:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:28:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:28:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:28:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:28:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:28:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_data_pool
2024-05-08T01:28:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_data_pool']
2024-05-08T01:28:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:28:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:28:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:28:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 1
2024-05-08T01:28:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:28:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:28:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_data_pool
2024-05-08T01:28:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:28:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:28:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:28:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:28:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:28:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:28:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:28:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:28:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:28:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:28:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:28:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:28:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:28:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:28:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_data_pool
2024-05-08T01:28:24.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_data_pool']
2024-05-08T01:28:24.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:28:24.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:28:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_data_pool
2024-05-08T01:28:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:28:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:28:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:28:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:28:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:28:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:28:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:28:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:28:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:28:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:28:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:28:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:28:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:28:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:28:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:28:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:28:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:28:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:28:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:28:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_data_pool
2024-05-08T01:28:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_data_pool']
2024-05-08T01:28:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:28:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:28:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test_data_pool
2024-05-08T01:28:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:28:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:28:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:28:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:28:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:28:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:28:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:28:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:28:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:28:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:28:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:28:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:28:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:28:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:28:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:28:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:28:24.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:28:24.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:28:24.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:28:24.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_data_pool
2024-05-08T01:28:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_data_pool']
2024-05-08T01:28:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:28:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:28:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test_data_pool
2024-05-08T01:28:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:28:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:28:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:28:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:28:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:28:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:28:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:28:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:28:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:28:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:28:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:28:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:28:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:28:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:28:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:28:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:28:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:28:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:28:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:28:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_data_pool
2024-05-08T01:28:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test_data_pool']
2024-05-08T01:28:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:28:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:28:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:28:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 2
2024-05-08T01:28:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:28:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:28:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_data_pool
2024-05-08T01:28:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:28:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:28:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:28:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:28:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:28:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:28:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:28:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:28:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:28:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:28:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:28:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:28:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:28:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:28:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:28:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:28:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:28:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:28:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:28:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:28:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:28:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:28:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:28:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:28:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_data_pool
2024-05-08T01:28:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:28:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:28:26.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:28:26.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:28:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:28:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:28:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:28:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_data_pool cluster1 mirror test_data_pool
2024-05-08T01:28:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 info mirror/test_data_pool
2024-05-08T01:28:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ awk '$1 == "data_pool:" {print $2}'
2024-05-08T01:28:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ data_pool=mirror_parent
2024-05-08T01:28:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test mirror_parent = mirror_parent
2024-05-08T01:28:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_snapshot cluster2 mirror test_data_pool snap1
2024-05-08T01:28:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap1
2024-05-08T01:28:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap create mirror/test_data_pool@snap1
2024-05-08T01:28:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Creating snap: 10% complete...
Creating snap: 100% complete...done.
2024-05-08T01:28:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test_data_pool 100
2024-05-08T01:28:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:28:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:28:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:28:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:28:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:28:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:28:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:28:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 1923.11   bytes/sec: 7.5 MiB/s
2024-05-08T01:28:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_snapshot cluster2 mirror test_data_pool snap2
2024-05-08T01:28:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap2
2024-05-08T01:28:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap create mirror/test_data_pool@snap2
2024-05-08T01:28:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Creating snap: 10% complete...
Creating snap: 100% complete...done.
2024-05-08T01:28:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test_data_pool 100
2024-05-08T01:28:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:28:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:28:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:28:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:28:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:28:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:28:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:28:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2083.37   bytes/sec: 8.1 MiB/s
2024-05-08T01:28:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_data_pool
2024-05-08T01:28:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:28:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:28:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test_data_pool
2024-05-08T01:28:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:28:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:28:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:28:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:28:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:28:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:28:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test_data_pool
2024-05-08T01:28:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:28:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test_data_pool ']'
2024-05-08T01:28:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test_data_pool'
2024-05-08T01:28:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool
2024-05-08T01:28:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:28:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:28:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:28:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:28:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:28:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:28:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool
2024-05-08T01:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:28:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:28:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:28:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:28:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:28:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:28:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:28:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:28:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:28:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:28:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:28:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:28:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:28:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:28:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:28:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:28:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool
2024-05-08T01:28:29.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test_data_pool
2024-05-08T01:28:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_data_pool ''
2024-05-08T01:28:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:28:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:28:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool
2024-05-08T01:28:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:28:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:28:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:28:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:28:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:28:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:28:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test_data_pool
2024-05-08T01:28:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_data_pool '..*'
2024-05-08T01:28:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:28:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:28:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool
2024-05-08T01:28:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:28:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:28:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:28:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:28:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:28:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:28:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:28:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' '!=' ''
2024-05-08T01:28:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:28:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' = ''
2024-05-08T01:28:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:28:29.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:28:29.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:28:29.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=5
2024-05-08T01:28:29.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:28:29.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:28:29.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=1
2024-05-08T01:28:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:28:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:28:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=101
2024-05-08T01:28:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:28:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:28:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=3
2024-05-08T01:28:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 5 = 1 -a 101 -le 3
2024-05-08T01:28:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:28:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:28:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:28:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test_data_pool
2024-05-08T01:28:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:28:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test_data_pool ']'
2024-05-08T01:28:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test_data_pool'
2024-05-08T01:28:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool
2024-05-08T01:28:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:28:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:28:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:28:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:28:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:28:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:28:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:28:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool
2024-05-08T01:28:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:28:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:28:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:28:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:28:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:28:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:28:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:28:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:28:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:28:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:28:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:28:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:28:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:28:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:28:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:28:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:28:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:28:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:28:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:28:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:28:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:28:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:28:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool
2024-05-08T01:28:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test_data_pool
2024-05-08T01:28:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_data_pool ''
2024-05-08T01:28:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:28:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:28:29.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool
2024-05-08T01:28:29.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:28:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:28:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:28:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:28:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2]]], state=connected]
2024-05-08T01:28:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:28:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test_data_pool
2024-05-08T01:28:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_data_pool '..*'
2024-05-08T01:28:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:28:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:28:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool
2024-05-08T01:28:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:28:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:28:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:28:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:28:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2]]], state=connected]
2024-05-08T01:28:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1'
2024-05-08T01:28:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=1'
2024-05-08T01:28:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:28:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:28:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:28:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=1, tag_tid=2, entry_tid=1'
2024-05-08T01:28:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:28:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:28:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=5
2024-05-08T01:28:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1'
2024-05-08T01:28:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:28:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=2
2024-05-08T01:28:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:28:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:28:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=101
2024-05-08T01:28:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1'
2024-05-08T01:28:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:28:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=1
2024-05-08T01:28:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 5 = 2 -a 101 -le 1
2024-05-08T01:28:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:28:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:28:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:28:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test_data_pool
2024-05-08T01:28:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:28:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test_data_pool ']'
2024-05-08T01:28:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test_data_pool'
2024-05-08T01:28:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool
2024-05-08T01:28:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:28:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:28:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:28:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:28:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:28:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:28:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:28:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool
2024-05-08T01:28:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:28:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:28:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:28:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:28:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:28:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:28:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:28:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:28:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:28:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:28:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:28:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:28:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:28:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:28:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:28:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:28:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:28:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:28:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:28:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:28:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:28:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:28:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool
2024-05-08T01:28:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test_data_pool
2024-05-08T01:28:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_data_pool ''
2024-05-08T01:28:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:28:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:28:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool
2024-05-08T01:28:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:28:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:28:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:28:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:28:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=2, tag_tid=3, entry_tid=14], [object_number=1, tag_tid=3, entry_tid=13], [object_number=0, tag_tid=3, entry_tid=12], [object_number=3, tag_tid=3, entry_tid=11]]], state=connected]
2024-05-08T01:28:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:28:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test_data_pool
2024-05-08T01:28:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_data_pool '..*'
2024-05-08T01:28:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:28:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:28:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool
2024-05-08T01:28:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:28:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:28:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:28:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:28:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=2, tag_tid=3, entry_tid=14], [object_number=1, tag_tid=3, entry_tid=13], [object_number=0, tag_tid=3, entry_tid=12], [object_number=3, tag_tid=3, entry_tid=11]]], state=connected]
2024-05-08T01:28:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=2, tag_tid=3, entry_tid=14'
2024-05-08T01:28:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=2, tag_tid=3, entry_tid=14'
2024-05-08T01:28:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=2, tag_tid=3, entry_tid=14' '!=' 'object_number=1, tag_tid=2, entry_tid=1'
2024-05-08T01:28:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:28:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=2, tag_tid=3, entry_tid=14' = 'object_number=1, tag_tid=2, entry_tid=1'
2024-05-08T01:28:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=2, tag_tid=3, entry_tid=14'
2024-05-08T01:28:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:28:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:28:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=5
2024-05-08T01:28:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=2, tag_tid=3, entry_tid=14'
2024-05-08T01:28:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:28:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=3
2024-05-08T01:28:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:28:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:28:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=101
2024-05-08T01:28:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=2, tag_tid=3, entry_tid=14'
2024-05-08T01:28:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:28:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=14
2024-05-08T01:28:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 5 = 3 -a 101 -le 14
2024-05-08T01:28:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:28:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:28:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:28:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test_data_pool
2024-05-08T01:28:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:28:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test_data_pool ']'
2024-05-08T01:28:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test_data_pool'
2024-05-08T01:28:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool
2024-05-08T01:28:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:28:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:28:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:28:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:28:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:28:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:28:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:28:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool
2024-05-08T01:28:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:28:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:28:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:28:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:28:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:28:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:28:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:28:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:28:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:28:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:28:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:28:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:28:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:28:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:28:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:28:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:28:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:30.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:28:30.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:30.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:28:30.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:28:30.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:28:30.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:28:30.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:28:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool
2024-05-08T01:28:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test_data_pool
2024-05-08T01:28:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_data_pool ''
2024-05-08T01:28:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:28:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:28:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool
2024-05-08T01:28:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:28:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:28:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:28:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:28:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=101], [object_number=0, tag_tid=3, entry_tid=100], [object_number=3, tag_tid=3, entry_tid=99], [object_number=2, tag_tid=3, entry_tid=98]]], state=connected]
2024-05-08T01:28:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:28:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test_data_pool
2024-05-08T01:28:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_data_pool '..*'
2024-05-08T01:28:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:28:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:28:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool
2024-05-08T01:28:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:28:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:28:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:28:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:28:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=101], [object_number=0, tag_tid=3, entry_tid=100], [object_number=3, tag_tid=3, entry_tid=99], [object_number=2, tag_tid=3, entry_tid=98]]], state=connected]
2024-05-08T01:28:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=101'
2024-05-08T01:28:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=3, entry_tid=101'
2024-05-08T01:28:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=1, tag_tid=3, entry_tid=101' '!=' 'object_number=2, tag_tid=3, entry_tid=14'
2024-05-08T01:28:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:28:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=1, tag_tid=3, entry_tid=101' = 'object_number=2, tag_tid=3, entry_tid=14'
2024-05-08T01:28:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=1, tag_tid=3, entry_tid=101'
2024-05-08T01:28:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:28:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:28:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=5
2024-05-08T01:28:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=101'
2024-05-08T01:28:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:28:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=3
2024-05-08T01:28:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:28:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:28:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=101
2024-05-08T01:28:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=101'
2024-05-08T01:28:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:28:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=101
2024-05-08T01:28:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 5 = 3 -a 101 -le 101
2024-05-08T01:28:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:28:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:28:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:28:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test_data_pool
2024-05-08T01:28:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:28:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test_data_pool ']'
2024-05-08T01:28:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test_data_pool'
2024-05-08T01:28:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool
2024-05-08T01:28:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:28:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:28:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:28:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:28:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:28:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:28:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:28:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool
2024-05-08T01:28:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:28:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:28:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:28:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:28:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:28:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:28:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:28:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:28:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:28:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:28:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:28:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:28:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:28:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:28:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:28:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:28:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:28:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:28:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:28:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:28:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:28:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:28:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool
2024-05-08T01:28:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test_data_pool
2024-05-08T01:28:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_data_pool ''
2024-05-08T01:28:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:28:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:28:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool
2024-05-08T01:28:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:28:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:28:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:28:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:28:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=2, tag_tid=5, entry_tid=2], [object_number=1, tag_tid=5, entry_tid=1], [object_number=0, tag_tid=5, entry_tid=0], [object_number=3, tag_tid=3, entry_tid=99]]], state=connected]
2024-05-08T01:28:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:28:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test_data_pool
2024-05-08T01:28:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_data_pool '..*'
2024-05-08T01:28:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:28:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:28:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool
2024-05-08T01:28:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:28:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:28:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:28:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:28:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=2, tag_tid=5, entry_tid=2], [object_number=1, tag_tid=5, entry_tid=1], [object_number=0, tag_tid=5, entry_tid=0], [object_number=3, tag_tid=3, entry_tid=99]]], state=connected]
2024-05-08T01:28:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=2, tag_tid=5, entry_tid=2'
2024-05-08T01:28:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=2, tag_tid=5, entry_tid=2'
2024-05-08T01:28:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=2, tag_tid=5, entry_tid=2' '!=' 'object_number=1, tag_tid=3, entry_tid=101'
2024-05-08T01:28:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:28:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=2, tag_tid=5, entry_tid=2' = 'object_number=1, tag_tid=3, entry_tid=101'
2024-05-08T01:28:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=2, tag_tid=5, entry_tid=2'
2024-05-08T01:28:31.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:28:31.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:28:31.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:31.437 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=5
2024-05-08T01:28:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=2, tag_tid=5, entry_tid=2'
2024-05-08T01:28:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:28:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=5
2024-05-08T01:28:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:28:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:28:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=101
2024-05-08T01:28:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=2, tag_tid=5, entry_tid=2'
2024-05-08T01:28:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:28:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:28:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=2
2024-05-08T01:28:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 5 = 5 -a 101 -le 2
2024-05-08T01:28:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:28:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:28:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:28:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test_data_pool
2024-05-08T01:28:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:28:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test_data_pool ']'
2024-05-08T01:28:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test_data_pool'
2024-05-08T01:28:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool
2024-05-08T01:28:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:28:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:28:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:28:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:28:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:28:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:28:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:28:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:28:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool
2024-05-08T01:28:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:28:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:28:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:28:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:28:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:28:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:28:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:28:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:28:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:28:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:28:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:28:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:28:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:28:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:28:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:28:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:28:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:28:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:28:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:28:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:28:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:28:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:28:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:28:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:28:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:28:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:28:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:28:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool
2024-05-08T01:28:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test_data_pool
2024-05-08T01:28:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_data_pool ''
2024-05-08T01:28:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:28:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:28:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool
2024-05-08T01:28:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:28:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:28:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:28:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:28:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:28:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:28:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test_data_pool
2024-05-08T01:28:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_data_pool '..*'
2024-05-08T01:28:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:28:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:28:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:28:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:28:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool
2024-05-08T01:28:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:28:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:28:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:28:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:28:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected]
2024-05-08T01:28:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.status
2024-05-08T01:28:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:28:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=5, entry_tid=101'
2024-05-08T01:28:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test_data_pool
2024-05-08T01:28:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:28:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_data_pool up+replaying primary_position
2024-05-08T01:28:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:28:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:28:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:28:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:28:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:28:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test_data_pool up+replaying primary_position ''
2024-05-08T01:28:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:28:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:28:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:28:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:28:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:28:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_data_pool
2024-05-08T01:28:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test_data_pool:
2024-05-08T01:28:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   150af226-2c33-4f60-805f-4fd4dadf2e2e
2024-05-08T01:28:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:28:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}
2024-05-08T01:28:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:28:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:28:24
2024-05-08T01:28:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:28:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:28:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:28:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:28:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:28:32'
2024-05-08T01:28:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:28:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:28:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:28:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}
2024-05-08T01:28:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:28:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:28:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:28:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:28:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:28:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:28:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:28:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:28:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:28:24'
2024-05-08T01:28:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_data_pool up+replaying '2024-05-08 01:28:24'
2024-05-08T01:28:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:28:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:28:24'
2024-05-08T01:28:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:28:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:28:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:28:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>2</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:118},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:118}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:28:04</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:28:32</last_update></peer_site></peer_sites></image><image><name>test_data_pool</name><global_id>150af226-2c33-4f60-805f-4fd4dadf2e2e</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:1}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:28:24</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:28:32</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:28:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:28:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/last_update'
2024-05-08T01:28:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:28:24'
2024-05-08T01:28:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/state'
2024-05-08T01:28:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:28:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:28:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:28:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:28:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror test_data_pool@snap1
2024-05-08T01:28:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool@snap1
2024-05-08T01:28:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:28:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror-test_data_pool@snap1.export
2024-05-08T01:28:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror-test_data_pool@snap1.export
2024-05-08T01:28:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:28:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool@snap1.export
2024-05-08T01:28:33.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror-test_data_pool@snap1.export
2024-05-08T01:28:33.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror-test_data_pool@snap1.export
2024-05-08T01:28:33.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:28:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror-test_data_pool@snap1.export
2024-05-08T01:28:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool@snap1.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test_data_pool@snap1.export
2024-05-08T01:28:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/test_data_pool@snap1 /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool@snap1.export
2024-05-08T01:28:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:28:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/test_data_pool@snap1 /tmp/tmp.3xddryFtVF/cluster1-mirror-test_data_pool@snap1.export
2024-05-08T01:28:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:28:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool@snap1.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test_data_pool@snap1.export
2024-05-08T01:28:33.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool@snap1.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test_data_pool@snap1.export
2024-05-08T01:28:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror test_data_pool@snap2
2024-05-08T01:28:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool@snap2
2024-05-08T01:28:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:28:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror-test_data_pool@snap2.export
2024-05-08T01:28:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror-test_data_pool@snap2.export
2024-05-08T01:28:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:28:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool@snap2.export
2024-05-08T01:28:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror-test_data_pool@snap2.export
2024-05-08T01:28:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror-test_data_pool@snap2.export
2024-05-08T01:28:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:28:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror-test_data_pool@snap2.export
2024-05-08T01:28:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool@snap2.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test_data_pool@snap2.export
2024-05-08T01:28:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/test_data_pool@snap2 /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool@snap2.export
2024-05-08T01:28:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:28:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/test_data_pool@snap2 /tmp/tmp.3xddryFtVF/cluster1-mirror-test_data_pool@snap2.export
2024-05-08T01:28:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:28:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool@snap2.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test_data_pool@snap2.export
2024-05-08T01:28:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool@snap2.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test_data_pool@snap2.export
2024-05-08T01:28:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror test_data_pool
2024-05-08T01:28:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:28:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror-test_data_pool.export
2024-05-08T01:28:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror-test_data_pool.export
2024-05-08T01:28:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:28:33.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.export
2024-05-08T01:28:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror-test_data_pool.export
2024-05-08T01:28:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror-test_data_pool.export
2024-05-08T01:28:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:28:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror-test_data_pool.export
2024-05-08T01:28:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test_data_pool.export
2024-05-08T01:28:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/test_data_pool /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.export
2024-05-08T01:28:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:28:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/test_data_pool /tmp/tmp.3xddryFtVF/cluster1-mirror-test_data_pool.export
2024-05-08T01:28:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:28:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test_data_pool.export
2024-05-08T01:28:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test_data_pool.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test_data_pool.export
2024-05-08T01:28:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster2 mirror test_data_pool
2024-05-08T01:28:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:28:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:28:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster2 mirror test_data_pool
2024-05-08T01:28:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:28:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_data_pool
2024-05-08T01:28:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing all snapshots: 50% complete...
Removing all snapshots: 100% complete...
Removing all snapshots: 100% complete...done.
2024-05-08T01:28:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rm mirror/test_data_pool
2024-05-08T01:28:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:28:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: disable mirroring / delete non-primary image'
2024-05-08T01:28:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:28:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:28:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:28:35 TEST: disable mirroring / delete non-primary image
2024-05-08T01:28:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:28:35 TEST: disable mirroring / delete non-primary image
2024-05-08T01:28:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image2=test2
2024-05-08T01:28:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image3=test3
2024-05-08T01:28:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image4=test4
2024-05-08T01:28:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image5=test5
2024-05-08T01:28:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in ${image2} ${image3} ${image4} ${image5}
2024-05-08T01:28:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image_and_enable_mirror cluster2 mirror test2 journal
2024-05-08T01:28:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:28:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:28:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n journal ']'
2024-05-08T01:28:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image cluster2 mirror test2
2024-05-08T01:28:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:28:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=128
2024-05-08T01:28:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:28:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test2
2024-05-08T01:28:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ is_pool_mirror_mode_image mirror
2024-05-08T01:28:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' pool = image ']'
2024-05-08T01:28:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ case "${pool}" in
2024-05-08T01:28:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:28:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test2 100
2024-05-08T01:28:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:28:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:28:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:28:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:28:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:28:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test2 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:28:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:28:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:28:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2272.76   bytes/sec: 8.9 MiB/s
2024-05-08T01:28:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_snapshot cluster2 mirror test2 snap1
2024-05-08T01:28:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:28:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap1
2024-05-08T01:28:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap create mirror/test2@snap1
2024-05-08T01:28:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Creating snap: 10% complete...
Creating snap: 100% complete...done.
2024-05-08T01:28:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_snapshot cluster2 mirror test2 snap2
2024-05-08T01:28:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:28:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap2
2024-05-08T01:28:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap create mirror/test2@snap2
2024-05-08T01:28:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Creating snap: 10% complete...
Creating snap: 100% complete...done.
2024-05-08T01:28:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' test2 = test4 ']'
2024-05-08T01:28:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' test2 = test5 ']'
2024-05-08T01:28:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test2 100
2024-05-08T01:28:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:28:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:28:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:28:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:28:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:28:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test2 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:28:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:28:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:28:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 1785.74   bytes/sec: 7.0 MiB/s
2024-05-08T01:28:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = snapshot ']'
2024-05-08T01:28:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test2 present
2024-05-08T01:28:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:28:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=present
2024-05-08T01:28:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:28:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:28:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:28:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test2
2024-05-08T01:28:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test2
2024-05-08T01:28:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test2
2024-05-08T01:28:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:28:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test2: (2) No such file or directory
2024-05-08T01:28:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=
2024-05-08T01:28:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:28:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:28:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test2 present ''
2024-05-08T01:28:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:28:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:28:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:28:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:28:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:28:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test2
2024-05-08T01:28:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test2
2024-05-08T01:28:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:28:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test2
2024-05-08T01:28:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=10de612a6ca8
2024-05-08T01:28:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 10de612a6ca8
2024-05-08T01:28:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z '' -o '' = 10de612a6ca8
2024-05-08T01:28:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:28:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = present
2024-05-08T01:28:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_snap_present cluster1 mirror test2 snap2
2024-05-08T01:28:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:28:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap_name=snap2
2024-05-08T01:28:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:28:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:28:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:28:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 info mirror/test2@snap2
2024-05-08T01:28:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:error setting snapshot context: (2) No such file or directory
2024-05-08T01:28:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ continue
2024-05-08T01:28:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:28:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:28:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 info mirror/test2@snap2
2024-05-08T01:28:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:rbd image 'test2':
2024-05-08T01:28:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	size 128 MiB in 32 objects
2024-05-08T01:28:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	order 22 (4 MiB objects)
2024-05-08T01:28:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	snapshot_count: 2
2024-05-08T01:28:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	id: 10de612a6ca8
2024-05-08T01:28:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	block_name_prefix: rbd_data.10de612a6ca8
2024-05-08T01:28:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	format: 2
2024-05-08T01:28:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	features: layering, exclusive-lock, journaling
2024-05-08T01:28:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	op_features:
2024-05-08T01:28:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	flags:
2024-05-08T01:28:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	create_timestamp: Wed May  8 01:28:37 2024
2024-05-08T01:28:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	access_timestamp: Wed May  8 01:28:37 2024
2024-05-08T01:28:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	modify_timestamp: Wed May  8 01:28:37 2024
2024-05-08T01:28:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	protected: False
2024-05-08T01:28:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	journal: 10de612a6ca8
2024-05-08T01:28:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring state: enabled
2024-05-08T01:28:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring mode: journal
2024-05-08T01:28:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring global id: 4671216e-1079-468e-9c4d-608b6681ebee
2024-05-08T01:28:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring primary: false
2024-05-08T01:28:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in ${image2} ${image3} ${image4} ${image5}
2024-05-08T01:28:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image_and_enable_mirror cluster2 mirror test3 journal
2024-05-08T01:28:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test3
2024-05-08T01:28:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:28:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n journal ']'
2024-05-08T01:28:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image cluster2 mirror test3
2024-05-08T01:28:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test3
2024-05-08T01:28:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=128
2024-05-08T01:28:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:28:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test3
2024-05-08T01:28:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ is_pool_mirror_mode_image mirror
2024-05-08T01:28:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' pool = image ']'
2024-05-08T01:28:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ case "${pool}" in
2024-05-08T01:28:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:28:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test3 100
2024-05-08T01:28:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test3
2024-05-08T01:28:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:28:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:28:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:28:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:28:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test3 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:28:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:28:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:28:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2083.36   bytes/sec: 8.1 MiB/s
2024-05-08T01:28:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_snapshot cluster2 mirror test3 snap1
2024-05-08T01:28:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test3
2024-05-08T01:28:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap1
2024-05-08T01:28:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap create mirror/test3@snap1
2024-05-08T01:28:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Creating snap: 10% complete...
Creating snap: 100% complete...done.
2024-05-08T01:28:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_snapshot cluster2 mirror test3 snap2
2024-05-08T01:28:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test3
2024-05-08T01:28:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap2
2024-05-08T01:28:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap create mirror/test3@snap2
2024-05-08T01:28:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Creating snap: 10% complete...
Creating snap: 100% complete...done.
2024-05-08T01:28:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' test3 = test4 ']'
2024-05-08T01:28:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' test3 = test5 ']'
2024-05-08T01:28:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test3 100
2024-05-08T01:28:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test3
2024-05-08T01:28:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:28:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:28:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:28:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:28:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test3 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:28:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:28:42.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:28:42.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2083.36   bytes/sec: 8.1 MiB/s
2024-05-08T01:28:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = snapshot ']'
2024-05-08T01:28:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test3 present
2024-05-08T01:28:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test3
2024-05-08T01:28:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=present
2024-05-08T01:28:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:28:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:28:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:28:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test3
2024-05-08T01:28:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test3
2024-05-08T01:28:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test3
2024-05-08T01:28:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:28:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test3: (2) No such file or directory
2024-05-08T01:28:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=
2024-05-08T01:28:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:28:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:28:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test3 present ''
2024-05-08T01:28:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test3
2024-05-08T01:28:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:28:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:28:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:28:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:28:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test3
2024-05-08T01:28:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test3
2024-05-08T01:28:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test3
2024-05-08T01:28:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:28:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test3: (2) No such file or directory
2024-05-08T01:28:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:28:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:28:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:28:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:28:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:28:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test3 present ''
2024-05-08T01:28:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test3
2024-05-08T01:28:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:28:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:28:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:28:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:28:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test3
2024-05-08T01:28:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test3
2024-05-08T01:28:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test3
2024-05-08T01:28:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:28:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=10de13024f1c
2024-05-08T01:28:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 10de13024f1c
2024-05-08T01:28:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z '' -o '' = 10de13024f1c
2024-05-08T01:28:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:28:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = present
2024-05-08T01:28:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_snap_present cluster1 mirror test3 snap2
2024-05-08T01:28:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test3
2024-05-08T01:28:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap_name=snap2
2024-05-08T01:28:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:28:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:28:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:28:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 info mirror/test3@snap2
2024-05-08T01:28:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:error setting snapshot context: (2) No such file or directory
2024-05-08T01:28:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ continue
2024-05-08T01:28:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:28:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:28:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 info mirror/test3@snap2
2024-05-08T01:28:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:rbd image 'test3':
2024-05-08T01:28:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	size 128 MiB in 32 objects
2024-05-08T01:28:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	order 22 (4 MiB objects)
2024-05-08T01:28:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	snapshot_count: 3
2024-05-08T01:28:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	id: 10de13024f1c
2024-05-08T01:28:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	block_name_prefix: rbd_data.10de13024f1c
2024-05-08T01:28:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	format: 2
2024-05-08T01:28:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	features: layering, exclusive-lock, journaling, operations
2024-05-08T01:28:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	op_features: snap-trash
2024-05-08T01:28:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	flags:
2024-05-08T01:28:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	create_timestamp: Wed May  8 01:28:43 2024
2024-05-08T01:28:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	access_timestamp: Wed May  8 01:28:43 2024
2024-05-08T01:28:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	modify_timestamp: Wed May  8 01:28:43 2024
2024-05-08T01:28:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	protected: False
2024-05-08T01:28:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	journal: 10de13024f1c
2024-05-08T01:28:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring state: enabled
2024-05-08T01:28:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring mode: journal
2024-05-08T01:28:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring global id: 947db88e-34f0-44cf-9329-ca207a8ca8b5
2024-05-08T01:28:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring primary: false
2024-05-08T01:28:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in ${image2} ${image3} ${image4} ${image5}
2024-05-08T01:28:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image_and_enable_mirror cluster2 mirror test4 journal
2024-05-08T01:28:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:28:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:28:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n journal ']'
2024-05-08T01:28:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image cluster2 mirror test4
2024-05-08T01:28:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:28:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=128
2024-05-08T01:28:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:28:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test4
2024-05-08T01:28:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ is_pool_mirror_mode_image mirror
2024-05-08T01:28:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' pool = image ']'
2024-05-08T01:28:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ case "${pool}" in
2024-05-08T01:28:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:28:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test4 100
2024-05-08T01:28:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:28:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:28:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:28:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:28:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:28:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test4 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:28:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:28:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:28:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2083.36   bytes/sec: 8.1 MiB/s
2024-05-08T01:28:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_snapshot cluster2 mirror test4 snap1
2024-05-08T01:28:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:28:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap1
2024-05-08T01:28:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap create mirror/test4@snap1
2024-05-08T01:28:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Creating snap: 10% complete...
Creating snap: 100% complete...done.
2024-05-08T01:28:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_snapshot cluster2 mirror test4 snap2
2024-05-08T01:28:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:28:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap2
2024-05-08T01:28:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap create mirror/test4@snap2
2024-05-08T01:28:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Creating snap: 10% complete...
Creating snap: 100% complete...done.
2024-05-08T01:28:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' test4 = test4 ']'
2024-05-08T01:28:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ protect_snapshot cluster2 mirror test4 snap1
2024-05-08T01:28:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:28:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap1
2024-05-08T01:28:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap protect mirror/test4@snap1
2024-05-08T01:28:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ protect_snapshot cluster2 mirror test4 snap2
2024-05-08T01:28:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:28:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap2
2024-05-08T01:28:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap protect mirror/test4@snap2
2024-05-08T01:28:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test4 100
2024-05-08T01:28:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:28:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:28:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:28:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:28:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:28:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test4 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:28:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:28:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:28:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 1923.11   bytes/sec: 7.5 MiB/s
2024-05-08T01:28:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = snapshot ']'
2024-05-08T01:28:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test4 present
2024-05-08T01:28:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:28:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=present
2024-05-08T01:28:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:28:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:28:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:28:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test4
2024-05-08T01:28:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test4
2024-05-08T01:28:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test4
2024-05-08T01:28:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:28:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test4: (2) No such file or directory
2024-05-08T01:28:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=
2024-05-08T01:28:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:28:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:28:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test4 present ''
2024-05-08T01:28:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:28:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:28:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:28:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:28:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:28:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test4
2024-05-08T01:28:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test4
2024-05-08T01:28:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test4
2024-05-08T01:28:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:28:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test4: (2) No such file or directory
2024-05-08T01:28:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:28:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:28:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:28:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:28:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:28:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test4 present ''
2024-05-08T01:28:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:28:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:28:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:28:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:28:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:28:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test4
2024-05-08T01:28:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test4
2024-05-08T01:28:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test4
2024-05-08T01:28:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:28:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=10de69b1684
2024-05-08T01:28:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 10de69b1684
2024-05-08T01:28:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z '' -o '' = 10de69b1684
2024-05-08T01:28:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:28:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = present
2024-05-08T01:28:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_snap_present cluster1 mirror test4 snap2
2024-05-08T01:28:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:28:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap_name=snap2
2024-05-08T01:28:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:28:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:28:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:28:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 info mirror/test4@snap2
2024-05-08T01:28:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:error setting snapshot context: (2) No such file or directory
2024-05-08T01:28:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ continue
2024-05-08T01:28:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:28:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:28:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 info mirror/test4@snap2
2024-05-08T01:28:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:rbd image 'test4':
2024-05-08T01:28:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	size 128 MiB in 32 objects
2024-05-08T01:28:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	order 22 (4 MiB objects)
2024-05-08T01:28:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	snapshot_count: 2
2024-05-08T01:28:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	id: 10de69b1684
2024-05-08T01:28:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	block_name_prefix: rbd_data.10de69b1684
2024-05-08T01:28:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	format: 2
2024-05-08T01:28:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	features: layering, exclusive-lock, journaling
2024-05-08T01:28:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	op_features:
2024-05-08T01:28:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	flags:
2024-05-08T01:28:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	create_timestamp: Wed May  8 01:28:49 2024
2024-05-08T01:28:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	access_timestamp: Wed May  8 01:28:49 2024
2024-05-08T01:28:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	modify_timestamp: Wed May  8 01:28:49 2024
2024-05-08T01:28:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	protected: True
2024-05-08T01:28:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	journal: 10de69b1684
2024-05-08T01:28:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring state: enabled
2024-05-08T01:28:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring mode: journal
2024-05-08T01:28:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring global id: 128f7bac-b1b4-46bc-ab9b-cd86e14e6660
2024-05-08T01:28:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring primary: false
2024-05-08T01:28:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in ${image2} ${image3} ${image4} ${image5}
2024-05-08T01:28:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image_and_enable_mirror cluster2 mirror test5 journal
2024-05-08T01:28:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:28:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:28:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n journal ']'
2024-05-08T01:28:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image cluster2 mirror test5
2024-05-08T01:28:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:28:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:28:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=128
2024-05-08T01:28:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:28:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test5
2024-05-08T01:28:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ is_pool_mirror_mode_image mirror
2024-05-08T01:28:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' pool = image ']'
2024-05-08T01:28:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ case "${pool}" in
2024-05-08T01:28:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:28:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test5 100
2024-05-08T01:28:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:28:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:28:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:28:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:28:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:28:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test5 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:28:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:28:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:28:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2272.76   bytes/sec: 8.9 MiB/s
2024-05-08T01:28:53.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_snapshot cluster2 mirror test5 snap1
2024-05-08T01:28:53.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:28:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap1
2024-05-08T01:28:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap create mirror/test5@snap1
2024-05-08T01:28:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Creating snap: 10% complete...
Creating snap: 100% complete...done.
2024-05-08T01:28:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_snapshot cluster2 mirror test5 snap2
2024-05-08T01:28:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:28:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap2
2024-05-08T01:28:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap create mirror/test5@snap2
2024-05-08T01:28:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Creating snap: 10% complete...
Creating snap: 100% complete...done.
2024-05-08T01:28:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' test5 = test4 ']'
2024-05-08T01:28:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' test5 = test5 ']'
2024-05-08T01:28:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ protect_snapshot cluster2 mirror test5 snap1
2024-05-08T01:28:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:28:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap1
2024-05-08T01:28:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap protect mirror/test5@snap1
2024-05-08T01:28:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ protect_snapshot cluster2 mirror test5 snap2
2024-05-08T01:28:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:28:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap2
2024-05-08T01:28:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap protect mirror/test5@snap2
2024-05-08T01:28:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test5 100
2024-05-08T01:28:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:28:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:28:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:28:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:28:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:28:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test5 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:28:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:28:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:28:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2272.76   bytes/sec: 8.9 MiB/s
2024-05-08T01:28:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = snapshot ']'
2024-05-08T01:28:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test5 present
2024-05-08T01:28:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:28:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=present
2024-05-08T01:28:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:28:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:28:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:28:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test5
2024-05-08T01:28:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test5
2024-05-08T01:28:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test5
2024-05-08T01:28:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:28:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test5: (2) No such file or directory
2024-05-08T01:28:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=
2024-05-08T01:28:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:28:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:28:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test5 present ''
2024-05-08T01:28:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:28:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:28:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:28:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:28:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:28:55.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test5
2024-05-08T01:28:55.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:55.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:55.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test5
2024-05-08T01:28:55.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test5
2024-05-08T01:28:55.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:28:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test5: (2) No such file or directory
2024-05-08T01:28:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:28:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:28:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:28:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:28:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:28:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test5 present ''
2024-05-08T01:28:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:28:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:28:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:28:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:28:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:28:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test5
2024-05-08T01:28:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:28:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:28:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test5
2024-05-08T01:28:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test5
2024-05-08T01:28:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:28:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=10dec4586338
2024-05-08T01:28:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 10dec4586338
2024-05-08T01:28:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z '' -o '' = 10dec4586338
2024-05-08T01:28:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:28:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = present
2024-05-08T01:28:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_snap_present cluster1 mirror test5 snap2
2024-05-08T01:28:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:28:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:28:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap_name=snap2
2024-05-08T01:28:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:28:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:28:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:28:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 info mirror/test5@snap2
2024-05-08T01:28:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:error setting snapshot context: (2) No such file or directory
2024-05-08T01:28:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ continue
2024-05-08T01:28:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:28:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:28:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 info mirror/test5@snap2
2024-05-08T01:28:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:rbd image 'test5':
2024-05-08T01:28:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	size 128 MiB in 32 objects
2024-05-08T01:28:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	order 22 (4 MiB objects)
2024-05-08T01:28:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	snapshot_count: 2
2024-05-08T01:28:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	id: 10dec4586338
2024-05-08T01:28:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	block_name_prefix: rbd_data.10dec4586338
2024-05-08T01:28:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	format: 2
2024-05-08T01:28:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	features: layering, exclusive-lock, journaling
2024-05-08T01:28:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	op_features:
2024-05-08T01:28:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	flags:
2024-05-08T01:28:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	create_timestamp: Wed May  8 01:28:55 2024
2024-05-08T01:28:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	access_timestamp: Wed May  8 01:28:55 2024
2024-05-08T01:28:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	modify_timestamp: Wed May  8 01:28:55 2024
2024-05-08T01:28:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	protected: True
2024-05-08T01:28:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	journal: 10dec4586338
2024-05-08T01:28:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring state: enabled
2024-05-08T01:28:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring mode: journal
2024-05-08T01:28:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring global id: 9da32115-3038-4dd7-9e79-1129b0817974
2024-05-08T01:28:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring primary: false
2024-05-08T01:28:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:28:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_pool_mirror_mode cluster2 mirror image
2024-05-08T01:28:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=image
2024-05-08T01:28:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image
2024-05-08T01:28:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:note: changing mirroring mode from pool to image
2024-05-08T01:28:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in ${image2} ${image4}
2024-05-08T01:28:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ disable_mirror cluster2 mirror test2
2024-05-08T01:28:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:28:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/test2
2024-05-08T01:28:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Mirroring disabled
2024-05-08T01:28:59.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in ${image2} ${image4}
2024-05-08T01:28:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ disable_mirror cluster2 mirror test4
2024-05-08T01:28:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:28:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/test4
2024-05-08T01:28:59.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Mirroring disabled
2024-05-08T01:28:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ unprotect_snapshot cluster2 mirror test5 snap1
2024-05-08T01:28:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:28:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap1
2024-05-08T01:28:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test5@snap1
2024-05-08T01:28:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ unprotect_snapshot cluster2 mirror test5 snap2
2024-05-08T01:28:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:28:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap2
2024-05-08T01:28:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test5@snap2
2024-05-08T01:28:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in ${image3} ${image5}
2024-05-08T01:28:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_snapshot cluster2 mirror test3 snap1
2024-05-08T01:28:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:28:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:28:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test3
2024-05-08T01:28:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap1
2024-05-08T01:28:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rm mirror/test3@snap1
2024-05-08T01:29:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing snap: 100% complete...done.
2024-05-08T01:29:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_snapshot cluster2 mirror test3 snap2
2024-05-08T01:29:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test3
2024-05-08T01:29:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap2
2024-05-08T01:29:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rm mirror/test3@snap2
2024-05-08T01:29:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing snap: 100% complete...done.
2024-05-08T01:29:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster2 mirror test3
2024-05-08T01:29:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test3
2024-05-08T01:29:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:29:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:29:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster2 mirror test3
2024-05-08T01:29:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test3
2024-05-08T01:29:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 snap purge mirror/test3
2024-05-08T01:29:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rm mirror/test3
2024-05-08T01:29:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:29:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in ${image3} ${image5}
2024-05-08T01:29:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_snapshot cluster2 mirror test5 snap1
2024-05-08T01:29:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:29:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap1
2024-05-08T01:29:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rm mirror/test5@snap1
2024-05-08T01:29:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing snap: 100% complete...done.
2024-05-08T01:29:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_snapshot cluster2 mirror test5 snap2
2024-05-08T01:29:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:29:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap2
2024-05-08T01:29:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rm mirror/test5@snap2
2024-05-08T01:29:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing snap: 100% complete...done.
2024-05-08T01:29:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster2 mirror test5
2024-05-08T01:29:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:29:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:29:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:29:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster2 mirror test5
2024-05-08T01:29:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:03.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:29:03.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 snap purge mirror/test5
2024-05-08T01:29:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rm mirror/test5
2024-05-08T01:29:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:29:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in ${image2} ${image3} ${image4} ${image5}
2024-05-08T01:29:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test2 deleted
2024-05-08T01:29:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:29:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=deleted
2024-05-08T01:29:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:29:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:29:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:29:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test2
2024-05-08T01:29:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test2
2024-05-08T01:29:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test2
2024-05-08T01:29:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=10de612a6ca8
2024-05-08T01:29:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:29:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:29:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test2 deleted 10de612a6ca8
2024-05-08T01:29:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:29:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:29:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=10de612a6ca8
2024-05-08T01:29:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:29:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:29:04.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test2
2024-05-08T01:29:04.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:04.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:04.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test2
2024-05-08T01:29:04.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test2
2024-05-08T01:29:04.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=10de612a6ca8
2024-05-08T01:29:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 10de612a6ca8
2024-05-08T01:29:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z 10de612a6ca8 -o 10de612a6ca8 = 10de612a6ca8
2024-05-08T01:29:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:29:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = present
2024-05-08T01:29:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:29:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:29:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test2 deleted 10de612a6ca8
2024-05-08T01:29:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:29:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:29:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=10de612a6ca8
2024-05-08T01:29:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:29:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:29:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test2
2024-05-08T01:29:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test2
2024-05-08T01:29:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test2
2024-05-08T01:29:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test2: (2) No such file or directory
2024-05-08T01:29:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:29:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:29:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = deleted
2024-05-08T01:29:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in ${image2} ${image3} ${image4} ${image5}
2024-05-08T01:29:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test3 deleted
2024-05-08T01:29:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test3
2024-05-08T01:29:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=deleted
2024-05-08T01:29:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:29:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:29:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:29:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test3
2024-05-08T01:29:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test3
2024-05-08T01:29:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test3
2024-05-08T01:29:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test3: (2) No such file or directory
2024-05-08T01:29:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=
2024-05-08T01:29:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:29:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:29:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test3 deleted ''
2024-05-08T01:29:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test3
2024-05-08T01:29:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:29:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:29:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:29:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:29:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test3
2024-05-08T01:29:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test3
2024-05-08T01:29:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test3
2024-05-08T01:29:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test3: (2) No such file or directory
2024-05-08T01:29:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:29:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:29:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = deleted
2024-05-08T01:29:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in ${image2} ${image3} ${image4} ${image5}
2024-05-08T01:29:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test4 deleted
2024-05-08T01:29:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:29:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=deleted
2024-05-08T01:29:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:29:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:29:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:29:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test4
2024-05-08T01:29:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test4
2024-05-08T01:29:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test4
2024-05-08T01:29:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test4: (2) No such file or directory
2024-05-08T01:29:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=
2024-05-08T01:29:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:29:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:29:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test4 deleted ''
2024-05-08T01:29:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:29:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:29:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:29:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:29:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:29:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test4
2024-05-08T01:29:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test4
2024-05-08T01:29:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test4
2024-05-08T01:29:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test4: (2) No such file or directory
2024-05-08T01:29:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:29:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:29:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = deleted
2024-05-08T01:29:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in ${image2} ${image3} ${image4} ${image5}
2024-05-08T01:29:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test5 deleted
2024-05-08T01:29:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:29:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=deleted
2024-05-08T01:29:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:29:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:29:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:29:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test5
2024-05-08T01:29:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test5
2024-05-08T01:29:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test5
2024-05-08T01:29:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test5: (2) No such file or directory
2024-05-08T01:29:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=
2024-05-08T01:29:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:29:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:29:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test5 deleted ''
2024-05-08T01:29:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test5
2024-05-08T01:29:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:29:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:29:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:29:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:29:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test5
2024-05-08T01:29:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test5
2024-05-08T01:29:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test5
2024-05-08T01:29:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test5: (2) No such file or directory
2024-05-08T01:29:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:29:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:29:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = deleted
2024-05-08T01:29:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:29:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_pool_mirror_mode cluster2 mirror pool
2024-05-08T01:29:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=pool
2024-05-08T01:29:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror pool
2024-05-08T01:29:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:note: changing mirroring mode from image to pool
2024-05-08T01:29:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in ${image2} ${image4}
2024-05-08T01:29:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ enable_journaling cluster2 mirror test2
2024-05-08T01:29:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:29:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 feature enable mirror/test2 journaling
2024-05-08T01:29:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test2 present
2024-05-08T01:29:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:29:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=present
2024-05-08T01:29:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:29:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:29:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:29:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test2
2024-05-08T01:29:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test2
2024-05-08T01:29:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test2
2024-05-08T01:29:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test2: (2) No such file or directory
2024-05-08T01:29:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=
2024-05-08T01:29:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:29:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:29:06.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test2 present ''
2024-05-08T01:29:06.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:29:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:29:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:29:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:29:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:29:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test2
2024-05-08T01:29:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test2
2024-05-08T01:29:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test2
2024-05-08T01:29:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test2: (2) No such file or directory
2024-05-08T01:29:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:29:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:29:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:29:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:29:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:29:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test2 present ''
2024-05-08T01:29:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:29:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:29:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:29:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:29:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:29:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test2
2024-05-08T01:29:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test2
2024-05-08T01:29:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test2
2024-05-08T01:29:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test2: (2) No such file or directory
2024-05-08T01:29:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:29:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:29:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:29:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:29:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:29:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test2 present ''
2024-05-08T01:29:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:29:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:29:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:29:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:29:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:29:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test2
2024-05-08T01:29:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test2
2024-05-08T01:29:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test2
2024-05-08T01:29:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test2: (2) No such file or directory
2024-05-08T01:29:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:29:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:29:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:29:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:29:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 4
2024-05-08T01:29:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test2 present ''
2024-05-08T01:29:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:29:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:29:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:29:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:29:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:29:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test2
2024-05-08T01:29:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test2
2024-05-08T01:29:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test2
2024-05-08T01:29:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=10ded2b61589
2024-05-08T01:29:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 10ded2b61589
2024-05-08T01:29:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z '' -o '' = 10ded2b61589
2024-05-08T01:29:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:29:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = present
2024-05-08T01:29:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_snap_present cluster1 mirror test2 snap2
2024-05-08T01:29:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:29:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap_name=snap2
2024-05-08T01:29:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:29:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:29:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:29:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 info mirror/test2@snap2
2024-05-08T01:29:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:rbd image 'test2':
2024-05-08T01:29:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	size 128 MiB in 32 objects
2024-05-08T01:29:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	order 22 (4 MiB objects)
2024-05-08T01:29:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	snapshot_count: 2
2024-05-08T01:29:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	id: 10ded2b61589
2024-05-08T01:29:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	block_name_prefix: rbd_data.10ded2b61589
2024-05-08T01:29:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	format: 2
2024-05-08T01:29:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	features: layering, exclusive-lock, journaling
2024-05-08T01:29:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	op_features:
2024-05-08T01:29:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	flags:
2024-05-08T01:29:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	create_timestamp: Wed May  8 01:29:13 2024
2024-05-08T01:29:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	access_timestamp: Wed May  8 01:29:13 2024
2024-05-08T01:29:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	modify_timestamp: Wed May  8 01:29:13 2024
2024-05-08T01:29:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	protected: False
2024-05-08T01:29:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	journal: 10ded2b61589
2024-05-08T01:29:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring state: enabled
2024-05-08T01:29:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring mode: journal
2024-05-08T01:29:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring global id: 913b5117-3b0e-441b-89b0-b86584f55335
2024-05-08T01:29:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring primary: false
2024-05-08T01:29:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test2
2024-05-08T01:29:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:29:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test2 started
2024-05-08T01:29:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:29:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:29:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:29:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:29:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test2 started
2024-05-08T01:29:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:29:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:29:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:29:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:29:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:29:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test2
2024-05-08T01:29:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:29:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:29:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:29:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:29:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:29:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:29:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:29:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test2
2024-05-08T01:29:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:29:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:29:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:29:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:29:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:29:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:29:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:29:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:29:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:29:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:29:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:29:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:29:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:29:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:29:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:29:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:29:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:29:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:29:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:29:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:29:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:29:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:29:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:29:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:29:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test2
2024-05-08T01:29:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:29:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:29:15.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:29:15.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:29:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:29:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:29:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:29:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test2
2024-05-08T01:29:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:29:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:29:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:29:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test2
2024-05-08T01:29:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:29:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:29:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:29:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:29:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:29:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:29:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:29:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test2
2024-05-08T01:29:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:29:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:29:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test2 ']'
2024-05-08T01:29:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test2'
2024-05-08T01:29:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test2
2024-05-08T01:29:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:29:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:29:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:29:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:29:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:29:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:29:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test2
2024-05-08T01:29:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:29:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:29:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:29:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:29:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:29:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:29:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:29:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:29:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:29:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:29:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:29:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:29:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test2
2024-05-08T01:29:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test2
2024-05-08T01:29:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test2
2024-05-08T01:29:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test2 ''
2024-05-08T01:29:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test2
2024-05-08T01:29:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:29:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test2.status
2024-05-08T01:29:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test2.status
2024-05-08T01:29:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:29:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test2.status
2024-05-08T01:29:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test2
2024-05-08T01:29:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test2.status
2024-05-08T01:29:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:29:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:29:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:29:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:29:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:29:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test2.status
2024-05-08T01:29:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:29:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test2
2024-05-08T01:29:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test2
2024-05-08T01:29:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test2 '..*'
2024-05-08T01:29:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test2
2024-05-08T01:29:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:29:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test2.status
2024-05-08T01:29:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test2.status
2024-05-08T01:29:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:29:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test2.status
2024-05-08T01:29:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test2
2024-05-08T01:29:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test2.status
2024-05-08T01:29:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:29:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:29:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:29:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:29:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:29:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test2.status
2024-05-08T01:29:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=1'
2024-05-08T01:29:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=3, tag_tid=1, entry_tid=3' -a 'object_number=3, tag_tid=1, entry_tid=3' = 'object_number=1, tag_tid=1, entry_tid=1'
2024-05-08T01:29:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=1' '!=' ''
2024-05-08T01:29:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:29:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=1' = ''
2024-05-08T01:29:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=1, tag_tid=1, entry_tid=1'
2024-05-08T01:29:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:29:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:29:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:29:16.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=1
2024-05-08T01:29:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=1'
2024-05-08T01:29:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:29:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:29:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=1
2024-05-08T01:29:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:29:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:29:16.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:29:16.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=3
2024-05-08T01:29:16.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=1'
2024-05-08T01:29:16.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:29:16.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:29:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=1
2024-05-08T01:29:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 1 = 1 -a 3 -le 1
2024-05-08T01:29:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:29:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:29:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:29:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test2
2024-05-08T01:29:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:29:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:29:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test2 ']'
2024-05-08T01:29:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test2'
2024-05-08T01:29:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test2
2024-05-08T01:29:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:29:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:29:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:29:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:29:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:29:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:29:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test2
2024-05-08T01:29:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:29:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:29:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:29:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:29:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:29:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:29:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:29:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:29:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:29:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:29:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:29:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:29:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:29:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:29:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:29:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:29:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test2
2024-05-08T01:29:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test2
2024-05-08T01:29:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test2
2024-05-08T01:29:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test2 ''
2024-05-08T01:29:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test2
2024-05-08T01:29:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:29:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test2.status
2024-05-08T01:29:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test2.status
2024-05-08T01:29:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:29:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test2.status
2024-05-08T01:29:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test2
2024-05-08T01:29:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test2.status
2024-05-08T01:29:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:29:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:29:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:29:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:29:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:29:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test2.status
2024-05-08T01:29:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:29:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test2
2024-05-08T01:29:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test2
2024-05-08T01:29:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test2 '..*'
2024-05-08T01:29:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test2
2024-05-08T01:29:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:29:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test2.status
2024-05-08T01:29:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test2.status
2024-05-08T01:29:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:29:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test2.status
2024-05-08T01:29:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test2
2024-05-08T01:29:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test2.status
2024-05-08T01:29:16.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:29:16.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:29:16.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:29:16.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:29:16.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:29:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test2.status
2024-05-08T01:29:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:29:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=3, tag_tid=1, entry_tid=3' -a 'object_number=3, tag_tid=1, entry_tid=3' = 'object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:29:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror test2
2024-05-08T01:29:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:29:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:29:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror-test2.export
2024-05-08T01:29:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror-test2.export
2024-05-08T01:29:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:29:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror-test2.export
2024-05-08T01:29:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror-test2.export
2024-05-08T01:29:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror-test2.export
2024-05-08T01:29:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:29:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror-test2.export
2024-05-08T01:29:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test2.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test2.export
2024-05-08T01:29:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/test2 /tmp/tmp.3xddryFtVF/cluster2-mirror-test2.export
2024-05-08T01:29:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:29:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/test2 /tmp/tmp.3xddryFtVF/cluster1-mirror-test2.export
2024-05-08T01:29:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:29:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror-test2.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test2.export
2024-05-08T01:29:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test2.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test2.export
2024-05-08T01:29:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in ${image2} ${image4}
2024-05-08T01:29:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ enable_journaling cluster2 mirror test4
2024-05-08T01:29:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:29:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 feature enable mirror/test4 journaling
2024-05-08T01:29:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test4 present
2024-05-08T01:29:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:29:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=present
2024-05-08T01:29:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:29:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:29:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:29:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test4
2024-05-08T01:29:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test4
2024-05-08T01:29:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test4
2024-05-08T01:29:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test4: (2) No such file or directory
2024-05-08T01:29:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=
2024-05-08T01:29:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:29:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:29:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test4 present ''
2024-05-08T01:29:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:29:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:29:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:29:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:29:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:29:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test4
2024-05-08T01:29:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test4
2024-05-08T01:29:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test4
2024-05-08T01:29:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test4: (2) No such file or directory
2024-05-08T01:29:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:29:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:29:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:29:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:29:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:29:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test4 present ''
2024-05-08T01:29:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:29:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:29:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:29:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:29:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:29:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test4
2024-05-08T01:29:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test4
2024-05-08T01:29:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test4
2024-05-08T01:29:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test4: (2) No such file or directory
2024-05-08T01:29:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:29:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:29:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:29:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:29:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:29:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test4 present ''
2024-05-08T01:29:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:29:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:29:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:29:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:29:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:29:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test4
2024-05-08T01:29:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test4
2024-05-08T01:29:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test4
2024-05-08T01:29:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:20.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=10deb94deda9
2024-05-08T01:29:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 10deb94deda9
2024-05-08T01:29:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z '' -o '' = 10deb94deda9
2024-05-08T01:29:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:29:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = present
2024-05-08T01:29:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_snap_present cluster1 mirror test4 snap2
2024-05-08T01:29:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:29:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap_name=snap2
2024-05-08T01:29:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:29:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:29:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:29:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 info mirror/test4@snap2
2024-05-08T01:29:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:rbd image 'test4':
2024-05-08T01:29:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	size 128 MiB in 32 objects
2024-05-08T01:29:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	order 22 (4 MiB objects)
2024-05-08T01:29:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	snapshot_count: 2
2024-05-08T01:29:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	id: 10deb94deda9
2024-05-08T01:29:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	block_name_prefix: rbd_data.10deb94deda9
2024-05-08T01:29:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	format: 2
2024-05-08T01:29:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	features: layering, exclusive-lock, journaling
2024-05-08T01:29:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	op_features:
2024-05-08T01:29:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	flags:
2024-05-08T01:29:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	create_timestamp: Wed May  8 01:29:19 2024
2024-05-08T01:29:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	access_timestamp: Wed May  8 01:29:19 2024
2024-05-08T01:29:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	modify_timestamp: Wed May  8 01:29:19 2024
2024-05-08T01:29:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	protected: False
2024-05-08T01:29:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	journal: 10deb94deda9
2024-05-08T01:29:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring state: enabled
2024-05-08T01:29:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring mode: journal
2024-05-08T01:29:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring global id: 06b34fe9-fde8-420d-8b6e-89f6d83ddc6a
2024-05-08T01:29:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring primary: false
2024-05-08T01:29:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test4
2024-05-08T01:29:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:29:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test4 started
2024-05-08T01:29:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:29:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:29:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:29:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:29:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test4 started
2024-05-08T01:29:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:29:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:29:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:29:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:29:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:29:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test4
2024-05-08T01:29:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:29:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:29:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:29:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:29:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:29:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:29:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:29:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test4
2024-05-08T01:29:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:29:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:29:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:29:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:29:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:29:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:29:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:29:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:29:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:29:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:29:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:29:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:29:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:29:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:29:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:29:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:29:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:29:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:29:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:29:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:29:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:29:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:29:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:29:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:29:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test4
2024-05-08T01:29:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:29:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:29:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:29:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:29:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:29:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:29:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:29:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test4
2024-05-08T01:29:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:29:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:29:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:29:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test4
2024-05-08T01:29:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:29:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:29:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:29:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:29:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:29:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:29:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:29:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test4
2024-05-08T01:29:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:29:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:29:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test4 ']'
2024-05-08T01:29:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test4'
2024-05-08T01:29:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test4
2024-05-08T01:29:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:29:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:29:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:29:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:29:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:29:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:29:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test4
2024-05-08T01:29:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:29:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:29:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:29:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:29:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:29:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:29:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:29:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:29:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:29:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:29:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:29:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:29:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:29:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:29:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:29:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:29:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test4
2024-05-08T01:29:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test4
2024-05-08T01:29:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test4
2024-05-08T01:29:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test4 ''
2024-05-08T01:29:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test4
2024-05-08T01:29:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:29:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test4.status
2024-05-08T01:29:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test4.status
2024-05-08T01:29:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:29:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test4.status
2024-05-08T01:29:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test4
2024-05-08T01:29:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test4.status
2024-05-08T01:29:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:29:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:29:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:29:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:29:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:29:23.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test4.status
2024-05-08T01:29:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:29:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test4
2024-05-08T01:29:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test4
2024-05-08T01:29:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test4 '..*'
2024-05-08T01:29:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test4
2024-05-08T01:29:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:29:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test4.status
2024-05-08T01:29:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test4.status
2024-05-08T01:29:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:29:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test4.status
2024-05-08T01:29:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test4
2024-05-08T01:29:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test4.status
2024-05-08T01:29:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:29:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:29:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:29:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:29:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:29:23.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test4.status
2024-05-08T01:29:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=1'
2024-05-08T01:29:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=3, tag_tid=1, entry_tid=3' -a 'object_number=3, tag_tid=1, entry_tid=3' = 'object_number=1, tag_tid=1, entry_tid=1'
2024-05-08T01:29:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=1' '!=' ''
2024-05-08T01:29:23.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:29:23.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=1' = ''
2024-05-08T01:29:23.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=1, tag_tid=1, entry_tid=1'
2024-05-08T01:29:23.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:29:23.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:29:23.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:29:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=1
2024-05-08T01:29:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=1'
2024-05-08T01:29:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:29:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:29:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=1
2024-05-08T01:29:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:29:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:29:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:29:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=3
2024-05-08T01:29:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=1'
2024-05-08T01:29:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:29:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:29:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=1
2024-05-08T01:29:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 1 = 1 -a 3 -le 1
2024-05-08T01:29:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:29:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:29:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:29:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test4
2024-05-08T01:29:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:29:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:29:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test4 ']'
2024-05-08T01:29:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test4'
2024-05-08T01:29:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test4
2024-05-08T01:29:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:29:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:29:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:29:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:29:23.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:29:23.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:29:23.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:23.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test4
2024-05-08T01:29:23.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:29:23.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:23.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:23.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:29:23.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:29:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:29:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:29:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:29:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:29:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:29:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:29:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:29:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:29:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:29:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:29:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:29:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:29:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:29:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test4
2024-05-08T01:29:23.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:23.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test4
2024-05-08T01:29:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test4
2024-05-08T01:29:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test4 ''
2024-05-08T01:29:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test4
2024-05-08T01:29:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:29:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test4.status
2024-05-08T01:29:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:29:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test4.status
2024-05-08T01:29:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test4.status
2024-05-08T01:29:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test4
2024-05-08T01:29:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test4.status
2024-05-08T01:29:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:29:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:29:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:29:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:29:23.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:29:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test4.status
2024-05-08T01:29:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:29:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test4
2024-05-08T01:29:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test4
2024-05-08T01:29:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test4 '..*'
2024-05-08T01:29:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test4
2024-05-08T01:29:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:29:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test4.status
2024-05-08T01:29:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test4.status
2024-05-08T01:29:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:29:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test4.status
2024-05-08T01:29:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test4
2024-05-08T01:29:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test4.status
2024-05-08T01:29:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:29:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:29:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:29:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:29:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:29:23.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test4.status
2024-05-08T01:29:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:29:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=3, tag_tid=1, entry_tid=3' -a 'object_number=3, tag_tid=1, entry_tid=3' = 'object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:29:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror test4
2024-05-08T01:29:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:29:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:29:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror-test4.export
2024-05-08T01:29:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror-test4.export
2024-05-08T01:29:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:29:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror-test4.export
2024-05-08T01:29:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror-test4.export
2024-05-08T01:29:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror-test4.export
2024-05-08T01:29:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:29:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror-test4.export
2024-05-08T01:29:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test4.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test4.export
2024-05-08T01:29:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/test4 /tmp/tmp.3xddryFtVF/cluster2-mirror-test4.export
2024-05-08T01:29:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:29:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/test4 /tmp/tmp.3xddryFtVF/cluster1-mirror-test4.export
2024-05-08T01:29:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:29:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror-test4.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test4.export
2024-05-08T01:29:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test4.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test4.export
2024-05-08T01:29:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: remove mirroring pool'
2024-05-08T01:29:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:29:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:29:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:29:24 TEST: remove mirroring pool
2024-05-08T01:29:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:29:24 TEST: remove mirroring pool
2024-05-08T01:29:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pool=pool_to_remove
2024-05-08T01:29:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2}
2024-05-08T01:29:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:29:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --cluster cluster1 osd pool create pool_to_remove 16 16
2024-05-08T01:29:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:pool 'pool_to_remove' already exists
2024-05-08T01:29:25.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:29:25.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 pool init pool_to_remove
2024-05-08T01:29:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 mirror pool enable pool_to_remove pool
2024-05-08T01:29:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2}
2024-05-08T01:29:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:29:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --cluster cluster2 osd pool create pool_to_remove 16 16
2024-05-08T01:29:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:pool 'pool_to_remove' already exists
2024-05-08T01:29:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:29:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 pool init pool_to_remove
2024-05-08T01:29:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 mirror pool enable pool_to_remove pool
2024-05-08T01:29:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ peer_add cluster1 pool_to_remove cluster2
2024-05-08T01:29:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=pool_to_remove
2024-05-08T01:29:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local client_cluster=cluster2
2024-05-08T01:29:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local remote_cluster=cluster2
2024-05-08T01:29:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local uuid_var_name
2024-05-08T01:29:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:29:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local error_code
2024-05-08T01:29:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local peer_uuid
2024-05-08T01:29:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:29:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:29:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool peer add pool_to_remove cluster2
2024-05-08T01:29:32.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ peer_uuid=6f506693-abee-4308-b8c9-6d8d14ed4ddb
2024-05-08T01:29:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=0
2024-05-08T01:29:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:29:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -eq 17 ']'
2024-05-08T01:29:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 0 -eq 0
2024-05-08T01:29:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:29:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ peer_add cluster2 pool_to_remove cluster1
2024-05-08T01:29:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=pool_to_remove
2024-05-08T01:29:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local client_cluster=cluster1
2024-05-08T01:29:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local remote_cluster=cluster1
2024-05-08T01:29:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local uuid_var_name
2024-05-08T01:29:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:29:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local error_code
2024-05-08T01:29:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local peer_uuid
2024-05-08T01:29:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:29:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:29:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 mirror pool peer add pool_to_remove cluster1
2024-05-08T01:29:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ peer_uuid=3e288a5e-22bf-4252-91d9-2917ec46024f
2024-05-08T01:29:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=0
2024-05-08T01:29:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:29:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -eq 17 ']'
2024-05-08T01:29:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 0 -eq 0
2024-05-08T01:29:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:29:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rdp_image=test_remove_data_pool
2024-05-08T01:29:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image cluster2 pool_to_remove test 128
2024-05-08T01:29:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=pool_to_remove
2024-05-08T01:29:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:29:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=128
2024-05-08T01:29:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 128 ']'
2024-05-08T01:29:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=128
2024-05-08T01:29:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling pool_to_remove/test
2024-05-08T01:29:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image cluster2 mirror test_remove_data_pool 128 --data-pool pool_to_remove
2024-05-08T01:29:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_remove_data_pool
2024-05-08T01:29:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=128
2024-05-08T01:29:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 128 ']'
2024-05-08T01:29:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=128
2024-05-08T01:29:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --data-pool pool_to_remove mirror/test_remove_data_pool
2024-05-08T01:29:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 pool_to_remove test 100
2024-05-08T01:29:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=pool_to_remove
2024-05-08T01:29:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:29:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:29:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:29:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:29:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:29:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench pool_to_remove/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:29:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:29:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:29:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2500.04   bytes/sec: 9.8 MiB/s
2024-05-08T01:29:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror test_remove_data_pool 100
2024-05-08T01:29:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_remove_data_pool
2024-05-08T01:29:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:29:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:29:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:29:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:29:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/test_remove_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:29:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:29:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:29:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2083.36   bytes/sec: 8.1 MiB/s
2024-05-08T01:29:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 pool_to_remove test
2024-05-08T01:29:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:29:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=pool_to_remove
2024-05-08T01:29:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:29:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:29:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 pool_to_remove test
2024-05-08T01:29:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:29:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=pool_to_remove
2024-05-08T01:29:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:29:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:29:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:29:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:29:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:29:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:29:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 pool_to_remove test
2024-05-08T01:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=pool_to_remove
2024-05-08T01:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush pool_to_remove/test'
2024-05-08T01:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush pool_to_remove/test
2024-05-08T01:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:29:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:29:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:29:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:29:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:29:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test
2024-05-08T01:29:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:29:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:29:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:29:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:29:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:29:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:29:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:29:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:29:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:29:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:29:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:29:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:29:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:29:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:29:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:29:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:29:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test
2024-05-08T01:29:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror flush :  flush rbd mirror
2024-05-08T01:29:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:1 rbd mirror flush pool_to_remove/test
2024-05-08T01:29:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:29:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:29:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:1
2024-05-08T01:29:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:29:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:29:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster1:1 ']'
2024-05-08T01:29:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:29:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:29:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:29:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:29:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:29:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:29:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:29:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:29:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:29:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:29:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:29:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:29:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:29:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:29:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:29:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:29:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:29:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush pool_to_remove/test
2024-05-08T01:29:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror flush :  flush rbd mirror
2024-05-08T01:29:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:2 rbd mirror flush pool_to_remove/test
2024-05-08T01:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:2
2024-05-08T01:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster1:2 ']'
2024-05-08T01:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:29:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:29:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:29:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:29:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:29:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:29:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:29:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:29:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:29:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:29:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:29:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:29:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush pool_to_remove/test
2024-05-08T01:29:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror flush :  flush rbd mirror
2024-05-08T01:29:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:3 rbd mirror flush pool_to_remove/test
2024-05-08T01:29:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:29:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:29:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:3
2024-05-08T01:29:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:29:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:29:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster1:3 ']'
2024-05-08T01:29:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:29:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:29:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:29:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:29:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:29:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:29:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:29:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:29:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:29:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:29:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:29:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:29:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:29:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:29:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:29:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:29:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:29:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush pool_to_remove/test
2024-05-08T01:29:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror flush :  flush rbd mirror
2024-05-08T01:29:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:29:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:29:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:29:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test
2024-05-08T01:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:29:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:29:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:29:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:29:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:29:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:29:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:29:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:29:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:29:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:29:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test
2024-05-08T01:29:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror flush :  flush rbd mirror
2024-05-08T01:29:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:1 rbd mirror flush pool_to_remove/test
2024-05-08T01:29:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:29:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:29:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:1
2024-05-08T01:29:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:29:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:29:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster1:1 ']'
2024-05-08T01:29:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:29:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:29:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:29:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:29:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:29:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:29:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:29:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:29:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:29:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:29:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:29:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:29:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:29:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:29:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:29:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:29:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:29:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush pool_to_remove/test
2024-05-08T01:29:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror flush :  flush rbd mirror
2024-05-08T01:29:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:2 rbd mirror flush pool_to_remove/test
2024-05-08T01:29:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:29:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:29:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:2
2024-05-08T01:29:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:29:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:29:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster1:2 ']'
2024-05-08T01:29:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:29:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:29:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:29:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:29:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:29:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:29:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:29:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:29:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:29:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:29:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:29:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:29:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:29:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:29:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:29:35.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:29:35.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:29:35.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush pool_to_remove/test
2024-05-08T01:29:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror flush :  flush rbd mirror
2024-05-08T01:29:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:3 rbd mirror flush pool_to_remove/test
2024-05-08T01:29:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:29:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:29:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:3
2024-05-08T01:29:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:29:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:29:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster1:3 ']'
2024-05-08T01:29:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:29:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:29:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:29:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:29:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:29:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:29:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:29:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:29:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:29:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:29:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:29:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:29:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:29:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:29:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:29:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:29:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:29:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush pool_to_remove/test
2024-05-08T01:29:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror flush :  flush rbd mirror
2024-05-08T01:29:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:29:37.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:29:37.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:29:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test
2024-05-08T01:29:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:29:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:29:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:29:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:29:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:29:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:29:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:29:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:29:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:29:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:29:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:29:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:29:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:29:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:29:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:29:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:29:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test
2024-05-08T01:29:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror flush :  flush rbd mirror
2024-05-08T01:29:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:1 rbd mirror flush pool_to_remove/test
2024-05-08T01:29:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:29:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:29:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:1
2024-05-08T01:29:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:29:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:29:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster1:1 ']'
2024-05-08T01:29:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:29:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:29:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:29:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:29:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:29:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:29:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:29:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:29:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:29:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:29:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:29:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:29:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:29:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:29:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:29:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:29:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:29:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush pool_to_remove/test
2024-05-08T01:29:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror flush :  flush rbd mirror
2024-05-08T01:29:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:2 rbd mirror flush pool_to_remove/test
2024-05-08T01:29:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:29:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:29:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:2
2024-05-08T01:29:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:29:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:29:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster1:2 ']'
2024-05-08T01:29:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:29:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:29:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:29:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:29:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:29:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:29:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:29:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:29:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:29:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:29:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:29:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:29:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:29:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:29:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:29:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:29:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:29:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush pool_to_remove/test
2024-05-08T01:29:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror flush :  flush rbd mirror
2024-05-08T01:29:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:3 rbd mirror flush pool_to_remove/test
2024-05-08T01:29:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:29:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:29:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:3
2024-05-08T01:29:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:29:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:29:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster1:3 ']'
2024-05-08T01:29:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:29:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:29:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:29:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:29:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:29:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:29:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:29:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:29:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:29:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:29:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:29:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:29:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:29:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:29:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:29:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:29:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:29:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush pool_to_remove/test
2024-05-08T01:29:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror flush :  flush rbd mirror
2024-05-08T01:29:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 4
2024-05-08T01:29:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:29:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:29:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test
2024-05-08T01:29:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:29:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:29:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:29:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:29:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:29:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:29:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:29:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:29:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:29:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:29:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:29:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:29:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:29:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:29:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:29:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:29:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test
2024-05-08T01:29:42.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 pool_to_remove test
2024-05-08T01:29:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=pool_to_remove
2024-05-08T01:29:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:29:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 pool_to_remove test ''
2024-05-08T01:29:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=pool_to_remove
2024-05-08T01:29:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:29:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:29:42.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:29:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image pool_to_remove/test
2024-05-08T01:29:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:29:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:29:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:29:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:29:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=f944f2ca-3d47-4b07-8af7-8a12e5b78ba1, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected]
2024-05-08T01:29:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:29:42.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 pool_to_remove test
2024-05-08T01:29:42.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:42.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=pool_to_remove
2024-05-08T01:29:42.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:29:42.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 pool_to_remove test '..*'
2024-05-08T01:29:42.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:42.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=pool_to_remove
2024-05-08T01:29:42.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:29:42.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:29:42.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:29:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image pool_to_remove/test
2024-05-08T01:29:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:29:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:29:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:29:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:29:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=f944f2ca-3d47-4b07-8af7-8a12e5b78ba1, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected]
2024-05-08T01:29:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1'
2024-05-08T01:29:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=1, tag_tid=2, entry_tid=1'
2024-05-08T01:29:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' ''
2024-05-08T01:29:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:29:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' = ''
2024-05-08T01:29:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=1, tag_tid=2, entry_tid=1'
2024-05-08T01:29:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:29:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:29:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:29:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=2
2024-05-08T01:29:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1'
2024-05-08T01:29:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:29:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:29:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=2
2024-05-08T01:29:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:29:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:29:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:29:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=3
2024-05-08T01:29:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1'
2024-05-08T01:29:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:29:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:29:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=1
2024-05-08T01:29:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 2 = 2 -a 3 -le 1
2024-05-08T01:29:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:29:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:29:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:29:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 pool_to_remove test
2024-05-08T01:29:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=pool_to_remove
2024-05-08T01:29:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:29:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:29:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:29:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush pool_to_remove/test'
2024-05-08T01:29:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush pool_to_remove/test
2024-05-08T01:29:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:29:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:29:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:29:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:29:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:29:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:29:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test
2024-05-08T01:29:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:29:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:29:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:29:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:29:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:29:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:29:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:29:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:29:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:29:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:29:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:29:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:29:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:29:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:29:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:29:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:29:42.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:42.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:42.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:42.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test
2024-05-08T01:29:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 pool_to_remove test
2024-05-08T01:29:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=pool_to_remove
2024-05-08T01:29:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:29:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 pool_to_remove test ''
2024-05-08T01:29:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=pool_to_remove
2024-05-08T01:29:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:29:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:29:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:29:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image pool_to_remove/test
2024-05-08T01:29:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:29:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:29:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:29:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:29:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=f944f2ca-3d47-4b07-8af7-8a12e5b78ba1, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:29:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:29:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 pool_to_remove test
2024-05-08T01:29:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=pool_to_remove
2024-05-08T01:29:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:29:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 pool_to_remove test '..*'
2024-05-08T01:29:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=pool_to_remove
2024-05-08T01:29:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:29:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:29:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:29:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image pool_to_remove/test
2024-05-08T01:29:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:29:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:29:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:29:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:29:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=f944f2ca-3d47-4b07-8af7-8a12e5b78ba1, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:29:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-pool_to_remove-test.status
2024-05-08T01:29:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 pool_to_remove test
2024-05-08T01:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=pool_to_remove
2024-05-08T01:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 pool_to_remove test up+replaying primary_position
2024-05-08T01:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=pool_to_remove
2024-05-08T01:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:29:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 pool_to_remove test up+replaying primary_position ''
2024-05-08T01:29:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=pool_to_remove
2024-05-08T01:29:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:29:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:29:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:29:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:29:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:29:43.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:29:43.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status pool_to_remove/test
2024-05-08T01:29:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:29:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   f06ece89-928e-40fd-95c9-f0bac81aca1d
2024-05-08T01:29:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:29:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:29:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:29:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:29:40
2024-05-08T01:29:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:29:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:29:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:29:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:29:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:29:43'
2024-05-08T01:29:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:29:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:29:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:29:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:29:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:29:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:29:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:29:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:29:43.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:29:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:29:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:29:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:29:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:29:40'
2024-05-08T01:29:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 pool_to_remove test up+replaying '2024-05-08 01:29:40'
2024-05-08T01:29:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=pool_to_remove
2024-05-08T01:29:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:29:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:29:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:29:40'
2024-05-08T01:29:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:29:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:29:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status pool_to_remove --verbose --format xml
2024-05-08T01:29:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>5819</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>5821</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>5822</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>5820</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>f06ece89-928e-40fd-95c9-f0bac81aca1d</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:1},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><daemon_service><service_id>4307</service_id><instance_id>5819</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:29:40</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>fc06ddd8-7a31-471e-bf7d-ba87ceda7ce0</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:29:43</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:29:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:29:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:29:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:29:40'
2024-05-08T01:29:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:29:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:29:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:29:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:29:44.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:29:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_remove_data_pool
2024-05-08T01:29:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:29:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_remove_data_pool
2024-05-08T01:29:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:29:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test_remove_data_pool
2024-05-08T01:29:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:29:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_remove_data_pool
2024-05-08T01:29:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:29:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:29:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:29:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:29:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:29:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror test_remove_data_pool
2024-05-08T01:29:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_remove_data_pool
2024-05-08T01:29:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:29:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test_remove_data_pool ']'
2024-05-08T01:29:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/test_remove_data_pool'
2024-05-08T01:29:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_remove_data_pool
2024-05-08T01:29:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:29:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:29:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:29:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:29:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:29:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:29:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_remove_data_pool
2024-05-08T01:29:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:29:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:29:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:29:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:29:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:29:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:29:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:29:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:29:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:29:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:29:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:29:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:29:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:29:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:29:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:29:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:29:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:29:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:29:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:29:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_remove_data_pool
2024-05-08T01:29:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror test_remove_data_pool
2024-05-08T01:29:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_remove_data_pool
2024-05-08T01:29:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_remove_data_pool ''
2024-05-08T01:29:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_remove_data_pool
2024-05-08T01:29:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:29:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_remove_data_pool.status
2024-05-08T01:29:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_remove_data_pool.status
2024-05-08T01:29:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:29:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_remove_data_pool.status
2024-05-08T01:29:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_remove_data_pool
2024-05-08T01:29:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_remove_data_pool.status
2024-05-08T01:29:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:29:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:29:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:29:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:29:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:29:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_remove_data_pool.status
2024-05-08T01:29:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:29:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror test_remove_data_pool
2024-05-08T01:29:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_remove_data_pool
2024-05-08T01:29:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror test_remove_data_pool '..*'
2024-05-08T01:29:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_remove_data_pool
2024-05-08T01:29:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:29:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-test_remove_data_pool.status
2024-05-08T01:29:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-test_remove_data_pool.status
2024-05-08T01:29:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:29:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-test_remove_data_pool.status
2024-05-08T01:29:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_remove_data_pool
2024-05-08T01:29:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-test_remove_data_pool.status
2024-05-08T01:29:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:29:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:29:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:29:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:29:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:29:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-test_remove_data_pool.status
2024-05-08T01:29:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:29:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:29:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test_remove_data_pool
2024-05-08T01:29:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_remove_data_pool
2024-05-08T01:29:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:29:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_remove_data_pool up+replaying primary_position
2024-05-08T01:29:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_remove_data_pool
2024-05-08T01:29:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:29:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:29:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:29:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:29:45.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test_remove_data_pool up+replaying primary_position ''
2024-05-08T01:29:45.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:45.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:45.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_remove_data_pool
2024-05-08T01:29:45.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:29:45.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:29:45.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:29:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:29:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:29:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_remove_data_pool
2024-05-08T01:29:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test_remove_data_pool:
2024-05-08T01:29:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   c2e3a90b-a45f-4833-8d7d-fafb3acaf584
2024-05-08T01:29:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:29:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:29:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:29:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:29:36
2024-05-08T01:29:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:29:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:29:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: up+stopped
2024-05-08T01:29:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: local image is primary
2024-05-08T01:29:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: 2024-05-08 01:29:42'
2024-05-08T01:29:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:29:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:29:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:29:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:29:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:29:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:29:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:29:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:29:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:29:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:29:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:29:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:29:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:29:36'
2024-05-08T01:29:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_remove_data_pool up+replaying '2024-05-08 01:29:36'
2024-05-08T01:29:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_remove_data_pool
2024-05-08T01:29:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:29:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:29:36'
2024-05-08T01:29:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:29:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:29:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:29:45.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>4</replaying></states></summary><daemons><daemon><service_id>4307</service_id><instance_id>4318</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>4311</service_id><instance_id>4323</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4317</service_id><instance_id>4327</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>4326</service_id><instance_id>4329</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>915c10af-6bd7-44b0-837b-a3e596bc8472</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:118},&quot;primary_position&quot;:{&quot;entry_tid&quot;:0,&quot;object_number&quot;:0,&quot;tag_tid&quot;:118}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:29:34</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:29:42</last_update></peer_site></peer_sites></image><image><name>test2</name><global_id>913b5117-3b0e-441b-89b0-b86584f55335</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:35.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.4,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:1},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:1}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:29:34</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:29:42</last_update></peer_site></peer_sites></image><image><name>test4</name><global_id>06b34fe9-fde8-420d-8b6e-89f6d83ddc6a</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:35.0,&quot;entries_behind_primary&quot;:0,&quot;entries_per_second&quot;:0.4,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:1},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:1}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:29:34</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:29:42</last_update></peer_site></peer_sites></image><image><name>test_remove_data_pool</name><global_id>c2e3a90b-a45f-4833-8d7d-fafb3acaf584</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:101,&quot;object_number&quot;:1,&quot;tag_tid&quot;:1},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><daemon_service><service_id>4307</service_id><instance_id>4318</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:29:36</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>up+stopped</state><description>local image is primary</description><last_update>2024-05-08 01:29:42</last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:29:45.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:29:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_remove_data_pool'\'']/last_update'
2024-05-08T01:29:45.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:29:36'
2024-05-08T01:29:45.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_remove_data_pool'\'']/state'
2024-05-08T01:29:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:29:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:29:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:29:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:29:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2}
2024-05-08T01:29:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:29:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --cluster cluster1 osd pool rm pool_to_remove pool_to_remove --yes-i-really-really-mean-it
2024-05-08T01:29:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:pool 'pool_to_remove' does not exist
2024-05-08T01:29:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2}
2024-05-08T01:29:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:29:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --cluster cluster2 osd pool rm pool_to_remove pool_to_remove --yes-i-really-really-mean-it
2024-05-08T01:29:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:pool 'pool_to_remove' does not exist
2024-05-08T01:29:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster2 mirror test_remove_data_pool
2024-05-08T01:29:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_remove_data_pool
2024-05-08T01:29:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:29:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:29:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster2 mirror test_remove_data_pool
2024-05-08T01:29:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_remove_data_pool
2024-05-08T01:29:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_remove_data_pool
2024-05-08T01:29:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rm mirror/test_remove_data_pool
2024-05-08T01:29:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 100% complete...done.
2024-05-08T01:29:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test_remove_data_pool deleted
2024-05-08T01:29:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_remove_data_pool
2024-05-08T01:29:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=deleted
2024-05-08T01:29:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:29:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:29:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:29:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test_remove_data_pool
2024-05-08T01:29:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_remove_data_pool
2024-05-08T01:29:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test_remove_data_pool
2024-05-08T01:29:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=3.10de9509892c
2024-05-08T01:29:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:29:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:29:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test_remove_data_pool deleted 3.10de9509892c
2024-05-08T01:29:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:29:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:29:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_remove_data_pool
2024-05-08T01:29:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:29:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=3.10de9509892c
2024-05-08T01:29:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:29:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:29:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test_remove_data_pool
2024-05-08T01:29:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:29:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:29:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_remove_data_pool
2024-05-08T01:29:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test_remove_data_pool
2024-05-08T01:29:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:29:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test_remove_data_pool: (2) No such file or directory
2024-05-08T01:29:48.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:29:48.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:29:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = deleted
2024-05-08T01:29:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in 0 1 2 4 8 8 8 8 16 16
2024-05-08T01:29:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:29:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror status pool_to_remove/test
2024-05-08T01:29:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:29:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:29:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:29:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:29:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:29:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:29:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:29:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:29:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:29:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:29:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:29:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:29:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:29:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:29:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:29:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:29:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:29:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:29:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:29:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:29:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:29:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:29:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:29:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:29:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:29:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:29:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:29:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:29:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:29:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:29:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:29:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:29:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:29:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:29:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:29:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:29:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:29:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:29:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:29:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:29:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:29:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:29:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:29:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:29:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:29:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:29:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:29:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:29:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:29:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:29:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:29:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:29:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:29:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:29:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:29:48.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:ERROR: (22) Invalid argument
2024-05-08T01:29:48.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:unknown command prefix rbd mirror status pool_to_remove/test
2024-05-08T01:29:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:29:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in 0 1 2 4 8 8 8 8 16 16
2024-05-08T01:29:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:29:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror status pool_to_remove/test
2024-05-08T01:29:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:29:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:29:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:29:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:29:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:29:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:29:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:29:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:29:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:29:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:29:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:29:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:29:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:29:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:29:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:29:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:29:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:29:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:29:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:29:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:29:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:29:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:29:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:29:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:29:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:29:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:29:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:29:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:29:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:29:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:29:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:29:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:29:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:29:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:29:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:29:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:29:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:29:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:29:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:29:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:29:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:29:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:29:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:29:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:29:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:29:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:29:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:29:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:29:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:29:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:29:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:29:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:29:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:29:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:29:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:29:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:29:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:29:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:29:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:29:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:29:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:29:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:29:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:29:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:29:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:29:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:29:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:29:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:29:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:29:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:29:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:29:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:29:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:29:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:29:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:29:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:29:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:29:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:29:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:29:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:29:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:29:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:29:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:29:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:29:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:29:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:29:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:29:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:29:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:29:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:29:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:29:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:29:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:29:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:29:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:29:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:29:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:29:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:29:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:29:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:29:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:29:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:29:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:29:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:29:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:29:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:29:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:29:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:29:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:29:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:29:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:29:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:29:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:29:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:29:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:29:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:29:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:29:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:29:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:29:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:29:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:29:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:29:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:29:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:29:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:29:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:29:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:29:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:29:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:29:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:29:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:29:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:29:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:29:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:29:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:29:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:29:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:29:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:29:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:29:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:29:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:29:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:29:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:29:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:29:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:29:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:29:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:29:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:29:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:29:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:29:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:29:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:29:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:29:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:29:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:29:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:29:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:29:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:29:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:29:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:29:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:29:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:29:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:29:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:29:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:29:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:29:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:29:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:29:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:29:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:29:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:29:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:29:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:29:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:29:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:29:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:29:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:29:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:29:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:29:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:29:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:29:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:29:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:29:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:29:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:29:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:29:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:29:51.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:51.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:29:51.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:29:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:29:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:29:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:29:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:29:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:29:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:29:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:29:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:29:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:29:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:29:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:29:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:29:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:29:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:29:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:29:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:29:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:29:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:29:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:29:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:29:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:29:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:29:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:29:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:29:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:29:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:29:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:29:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:29:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:29:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:29:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:29:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:29:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:29:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:29:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:29:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:29:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:29:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:29:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:29:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:29:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:29:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:29:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:29:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:29:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:29:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:29:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:29:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:29:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:29:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:29:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:29:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:29:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:29:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:29:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:29:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:29:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:29:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:29:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:29:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:29:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:29:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:29:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:29:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:29:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:29:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:29:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:29:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:29:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:29:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:29:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:29:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:29:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:29:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:29:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:29:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:29:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:29:53.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:53.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:29:53.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:53.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:53.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:29:53.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:29:53.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:29:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:29:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:29:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:29:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:29:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:29:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:29:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:29:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:29:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:29:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:29:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:29:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:29:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:29:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:29:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:29:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:29:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:29:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:29:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:29:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:29:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:29:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:29:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:29:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:29:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:29:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:29:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:29:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:29:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:29:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:29:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:29:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:29:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:29:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:29:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:29:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:29:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:29:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:29:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:29:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:29:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:29:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:29:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:29:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:29:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:29:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:29:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:29:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:29:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:29:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 4
2024-05-08T01:29:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:29:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:29:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:29:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:29:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:29:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:29:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:29:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:29:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:29:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:29:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:29:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:29:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:29:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:29:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:29:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:29:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:29:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:29:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:29:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:29:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:29:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:29:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:29:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:29:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:29:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:29:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:29:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:29:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:29:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:29:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:29:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:29:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:29:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:29:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:29:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:29:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:29:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:29:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:29:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:29:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:29:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:29:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:29:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:29:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:29:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:29:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:29:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:29:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:29:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:29:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:29:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:29:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:29:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:29:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:29:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:29:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:29:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:29:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:29:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:29:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:29:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:29:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:29:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:29:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:29:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:29:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:29:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:29:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:29:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:29:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:29:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:29:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:29:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:29:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:29:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:29:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:29:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:29:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:58.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:29:58.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:29:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:29:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:29:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:29:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:29:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:29:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:29:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:29:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:29:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:29:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:29:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:29:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:29:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:29:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:29:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:29:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:29:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:29:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:29:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:29:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:29:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:29:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:29:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:29:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:29:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:29:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:29:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:29:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:29:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:29:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:29:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:29:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:29:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:29:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:29:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:29:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:29:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:29:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:29:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 8
2024-05-08T01:30:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:30:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:30:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:30:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:30:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:30:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:30:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:30:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:30:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:30:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:30:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:30:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:30:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:30:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:30:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:30:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:30:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:30:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:30:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:30:06.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:06.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:30:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:30:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:30:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:30:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:30:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:30:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:30:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:30:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:30:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:30:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:30:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:30:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:30:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:30:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:30:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:30:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:30:06.786 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:30:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:30:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:30:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:30:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:30:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:30:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:30:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:30:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:30:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:30:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:30:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:30:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:30:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:30:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:30:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:30:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:30:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:30:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:30:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:30:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:30:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:30:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:30:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:30:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:30:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:30:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:30:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:30:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:30:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:30:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:30:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:30:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:30:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:30:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:30:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 8
2024-05-08T01:30:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:30:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:30:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:30:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:30:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:30:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:30:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:30:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:30:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:30:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:30:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:30:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:30:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:30:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:30:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:30:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:30:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:30:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:30:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:30:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:30:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:30:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:30:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:30:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:30:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:30:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:30:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:30:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:30:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:30:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:30:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:30:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:30:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:30:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:30:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:30:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:30:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:30:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:30:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:30:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:30:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:30:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:30:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:30:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:30:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:30:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:30:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:30:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:30:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:30:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:30:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:30:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:30:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:30:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:30:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:30:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:30:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:30:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:30:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:30:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:30:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:30:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:30:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:30:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:30:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:30:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:30:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:30:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:30:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:30:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:30:15.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:30:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 8
2024-05-08T01:30:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:30:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:30:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:30:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:30:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:30:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:30:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:30:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:30:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:30:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:30:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:30:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:30:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:30:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:30:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:30:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:30:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:30:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:30:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:30:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:30:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:30:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:30:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:30:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:30:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:30:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:30:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:30:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:30:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:30:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:30:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:30:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:30:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:30:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:30:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:30:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:30:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:30:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:30:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:30:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:30:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:30:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:30:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:30:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:30:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:30:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:30:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:30:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:24.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:30:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:30:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:30:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:30:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:30:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:30:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:30:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:30:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:30:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:30:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 8
2024-05-08T01:30:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:30:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:30:32.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:32.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:30:32.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:30:32.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:32.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:32.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:30:32.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:30:32.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:30:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:30:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:30:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:30:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:30:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:30:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:30:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:30:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:30:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:30:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:30:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:30:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:30:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:30:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:30:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:30:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:30:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:30:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:30:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:30:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:30:32.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:30:32.353 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:30:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:30:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:30:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:30:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:30:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:30:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:30:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:30:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:30:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:30:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:30:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:30:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:30:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:30:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:30:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:30:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:30:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:30:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:30:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:30:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:30:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:30:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:30:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:30:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:30:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:30:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:30:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:30:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:30:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:30:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:30:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:30:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:30:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:30:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:30:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:30:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 8
2024-05-08T01:30:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:30:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:30:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:30:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:30:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:30:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:30:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:30:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:30:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:30:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:30:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:30:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:30:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:30:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:30:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:30:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:30:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:30:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:30:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:30:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:30:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:30:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:30:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:30:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:30:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:30:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:30:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:30:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:30:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:30:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:30:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:30:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:30:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:30:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:30:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:30:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:30:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:30:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:30:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:30:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:30:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:30:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:30:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:30:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:30:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:30:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:30:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:30:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:30:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:30:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:30:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:30:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:30:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:30:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:30:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:30:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:30:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:30:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:41.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:41.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:41.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:30:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 8
2024-05-08T01:30:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:30:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:30:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:30:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:30:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:30:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:30:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:30:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:30:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:30:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:30:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:30:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:30:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:30:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:30:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:30:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:30:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:30:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:30:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:30:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:49.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:30:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:30:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:30:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:30:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:30:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:30:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:30:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:30:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:30:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:30:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:30:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:30:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:30:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:30:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:30:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:30:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:30:49.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:30:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:30:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:30:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:30:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:30:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:30:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:30:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:30:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:30:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:30:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:30:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:30:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:30:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:30:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:30:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:30:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:30:49.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:30:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:30:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:30:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:30:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:30:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:30:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:30:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:30:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:30:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:30:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:30:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:30:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:30:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:30:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:30:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:30:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:30:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:30:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 8
2024-05-08T01:30:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:30:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:30:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:30:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:30:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:30:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:30:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:30:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:30:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:30:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:30:57.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:30:57.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:30:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:30:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:30:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:30:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:30:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:30:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:30:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:30:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:30:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:30:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:30:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:30:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:30:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:30:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:30:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:30:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:30:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:30:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:30:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:30:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:30:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:30:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:30:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:30:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:30:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:30:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:30:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:30:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:30:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:30:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:30:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:30:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:30:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:30:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:30:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:30:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:30:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:30:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:30:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:30:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:30:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:30:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:30:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:58.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:58.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:30:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:30:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:30:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:30:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:30:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:30:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:30:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:30:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:30:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:30:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:30:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:30:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:30:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:30:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:30:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:30:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:30:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:30:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:30:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:30:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:30:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:30:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:30:58.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:30:58.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:30:58.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:30:58.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:30:58.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:30:58.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:30:58.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:30:58.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:30:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:30:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:30:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:30:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:30:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:30:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:30:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 8
2024-05-08T01:31:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:31:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:31:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:31:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:31:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:31:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:31:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:31:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:31:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:31:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:31:06.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:31:06.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:31:06.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:06.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:06.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:31:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:31:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:31:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:31:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:31:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:31:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:31:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:31:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:31:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:31:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:31:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:31:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:31:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:31:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:31:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:31:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:31:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:31:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:31:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:31:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:31:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:31:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:31:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:31:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:31:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:31:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:31:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:31:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:31:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:31:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:31:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:31:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:31:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:31:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:31:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:31:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:31:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:31:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:31:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:31:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:31:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:31:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:31:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:31:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:31:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:31:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:31:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:31:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:31:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:31:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:31:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:31:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:31:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:31:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:31:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:31:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:31:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:31:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:31:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 16
2024-05-08T01:31:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:31:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:31:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:31:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:31:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:31:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:31:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:31:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:31:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:31:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:31:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:31:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:31:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:31:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:23.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:31:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:31:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:31:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:31:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:31:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:31:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:31:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:31:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:31:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:31:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:31:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:31:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:31:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:31:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:31:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:31:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:31:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:31:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:31:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:31:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:31:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:31:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:31:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:31:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:31:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:31:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:31:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:31:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:31:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:31:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:31:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:31:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:31:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:31:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:23.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:31:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:31:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:31:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:31:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:31:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:31:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:31:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:31:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:31:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:31:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:31:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:31:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:31:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:31:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:31:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:31:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:31:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:31:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 16
2024-05-08T01:31:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:31:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:31:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:31:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:31:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:31:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:31:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:31:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:31:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:31:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:31:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:31:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:31:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:31:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:31:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:31:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:31:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:31:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:31:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:31:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:31:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:31:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:31:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:31:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:31:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:31:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:31:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:31:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:31:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:31:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:31:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:31:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:31:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:31:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:31:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:31:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:31:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:31:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:31:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:31:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:31:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:31:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:31:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:31:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:31:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:31:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:31:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:31:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:31:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:31:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:31:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:31:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:31:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:31:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:31:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:31:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:31:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:31:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:31:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:31:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:31:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:31:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:31:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:31:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:31:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:31:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:31:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:31:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:31:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:31:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:31:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:31:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:31:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster2 rbd mirror status pool_to_remove/test
2024-05-08T01:31:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster2
2024-05-08T01:31:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:31:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster2
2024-05-08T01:31:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:31:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:31:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:31:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:31:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:31:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:31:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:31:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:31:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:31:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:31:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:31:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:31:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:31:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:31:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:31:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:31:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:31:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:31:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:31:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:31:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:31:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:31:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:31:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:31:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:31:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:31:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:31:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:31:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:31:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:31:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:31:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:31:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:31:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:31:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:31:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:31:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:31:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:31:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:31:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:31:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:31:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:31:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:31:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:31:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:31:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:31:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:31:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:31:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:31:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:31:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:31:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:31:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:31:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:31:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:31:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:31:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:40.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:40.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:40.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:31:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:31:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:31:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:31:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:31:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:31:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:31:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:31:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:31:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:31:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:31:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:31:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:31:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:31:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:31:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:31:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:31:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:31:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:31:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:31:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:31:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:31:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:31:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:31:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:31:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:31:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:31:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:31:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:31:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:31:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:31:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:31:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:31:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:31:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:31:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:31:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:31:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:31:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:31:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:31:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:31:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:31:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:31:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:31:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:31:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:31:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:31:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:31:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:31:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:31:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:31:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:31:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:31:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:31:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:31:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:31:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:31:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:31:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:31:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:31:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:31:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:31:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:31:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:31:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:31:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:31:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:31:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:31:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:31:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:31:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:31:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:31:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:31:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:31:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:31:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:31:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:31:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:31:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:31:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:31:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:31:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:31:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:31:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:31:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:31:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:31:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:31:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:31:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:31:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:31:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:31:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:31:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:31:44.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:44.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:31:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:31:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:31:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:31:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:31:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:31:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:31:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:31:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:31:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:31:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:31:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:31:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:31:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:31:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:31:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:31:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:31:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:31:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:31:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:31:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:31:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:31:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:31:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:31:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:31:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:31:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:31:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:31:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:31:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:31:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:31:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:31:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:31:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:31:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:44.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:44.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:31:44.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:31:44.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:44.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:44.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:31:44.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:31:44.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:44.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:44.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:44.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:31:44.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:31:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:31:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:31:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:31:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:31:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:31:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:31:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:31:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:31:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:31:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:31:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:31:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:31:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:31:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:31:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:31:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:31:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:31:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:31:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:31:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:31:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:31:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:31:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:31:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:31:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:31:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:31:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:31:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:31:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:31:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 4
2024-05-08T01:31:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:31:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:31:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:31:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:31:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:31:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:31:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:31:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:31:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:31:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:31:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:31:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:31:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:31:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:31:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:31:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:31:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:31:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:31:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:31:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:31:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:31:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:31:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:31:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:31:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:31:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:31:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:31:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:31:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:31:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:31:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:31:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:31:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:31:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:31:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:31:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:31:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:31:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:31:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:31:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:31:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:31:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:31:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:31:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:31:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:31:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:31:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:31:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:31:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:31:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:31:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:31:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:31:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:31:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:31:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:31:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:31:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:31:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:31:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:31:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:31:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:31:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:31:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:31:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:31:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:31:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:31:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:31:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:31:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:31:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:31:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:49.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:31:49.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 8
2024-05-08T01:31:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:31:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:31:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:31:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:31:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:31:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:31:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:31:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:31:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:31:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:31:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:31:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:31:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:31:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:31:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:31:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:31:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:31:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:31:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:31:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:31:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:31:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:31:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:31:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:31:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:31:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:31:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:31:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:31:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:31:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:31:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:31:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:31:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:31:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:31:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:31:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:31:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:31:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:31:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:31:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:31:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:31:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:31:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:31:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:31:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:31:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:31:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:31:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:31:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:31:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:31:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:31:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:31:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:31:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:31:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:31:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:31:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:31:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:31:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:31:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:31:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:31:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:31:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:31:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:31:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:31:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:31:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:31:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:31:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:31:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:31:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:31:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:31:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:31:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:31:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:31:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:31:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:31:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:31:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:31:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:31:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:31:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:31:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:31:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:31:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:31:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:31:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:31:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:31:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:31:57.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:31:57.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:31:57.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:31:57.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:31:57.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 8
2024-05-08T01:32:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:32:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:32:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:32:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:32:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:32:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:32:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:32:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:32:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:32:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:32:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:32:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:32:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:32:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:32:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:32:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:32:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:32:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:32:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:32:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:32:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:32:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:32:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:32:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:32:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:32:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:32:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:32:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:32:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:32:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:32:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:32:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:32:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:32:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:32:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:32:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:32:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:32:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:32:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:32:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:32:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:32:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:32:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:32:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:32:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:32:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:32:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:32:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:32:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:32:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:32:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:32:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:32:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:32:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:32:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:32:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:32:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:32:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:32:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:32:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:32:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:32:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:32:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:32:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:32:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:32:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:32:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:32:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:32:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:32:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:32:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:06.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:32:06.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 8
2024-05-08T01:32:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:32:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:32:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:32:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:32:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:32:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:32:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:32:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:32:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:32:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:32:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:32:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:32:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:32:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:32:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:32:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:32:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:32:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:32:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:32:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:32:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:32:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:32:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:32:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:32:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:32:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:32:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:32:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:32:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:32:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:32:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:32:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:32:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:32:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:32:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:32:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:32:14.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:14.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:14.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:14.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:32:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:32:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:32:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:32:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:32:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:32:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:32:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:32:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:32:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:32:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:32:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:32:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:32:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:32:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:32:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:32:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:32:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:32:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:32:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:32:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:32:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:32:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:32:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:32:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:32:14.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:32:14.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:32:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:32:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:32:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:32:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:32:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:32:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:32:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:32:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:32:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 8
2024-05-08T01:32:22.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:32:22.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:32:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:32:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:32:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:32:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:32:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:32:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:32:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:32:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:32:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:32:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:32:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:32:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:32:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:32:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:32:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:32:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:32:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:32:22.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:22.688 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:32:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:32:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:32:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:32:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:32:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:32:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:32:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:32:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:32:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:32:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:32:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:32:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:32:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:32:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:32:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:32:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:32:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:22.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:22.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:22.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:22.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:22.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:22.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:32:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:32:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:32:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:32:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:32:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:32:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:32:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:32:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:32:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:32:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:32:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:32:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:32:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:32:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:32:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:32:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:32:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:22.966 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:32:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:32:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:32:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:32:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:32:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:32:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:32:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:32:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:32:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:32:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:32:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 8
2024-05-08T01:32:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:32:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:32:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:32:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:32:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:32:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:32:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:32:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:32:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:31.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:32:31.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:32:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:32:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:32:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:32:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:32:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:32:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:32:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:32:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:32:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:32:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:32:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:32:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:32:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:32:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:32:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:32:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:32:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:32:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:32:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:32:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:32:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:32:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:32:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:32:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:32:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:32:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:32:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:32:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:32:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:32:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:32:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:32:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:32:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:32:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:32:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:32:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:32:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:32:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:32:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:32:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:32:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:32:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:32:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:32:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:32:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:32:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:32:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:32:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:32:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:32:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:32:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:32:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:32:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:32:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:32:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:32:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:32:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:32:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:32:31.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:32:31.618 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:32:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:32:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 8
2024-05-08T01:32:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:32:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:32:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:32:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:32:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:32:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:32:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:32:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:32:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:32:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:32:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:32:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:32:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:32:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:32:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:32:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:32:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:32:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:32:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:32:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:32:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:32:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:32:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:32:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:32:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:32:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:32:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:32:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:32:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:32:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:32:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:32:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:32:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:32:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:32:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:32:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:32:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:32:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:32:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:32:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:32:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:32:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:32:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:32:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:32:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:32:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:32:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:32:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:32:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:32:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:32:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:32:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:32:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:32:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:32:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:32:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:32:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:32:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:32:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:32:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:32:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:32:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:32:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:32:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:32:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:32:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:32:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:32:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:32:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:32:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:32:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:32:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 8
2024-05-08T01:32:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:32:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:32:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:32:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:32:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:32:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:32:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:32:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:32:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:32:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:32:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:32:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:32:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:32:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:32:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:32:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:32:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:32:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:32:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:32:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:32:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:32:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:32:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:32:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:32:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:32:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:32:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:32:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:32:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:32:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:32:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:32:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:32:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:32:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:32:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:32:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:32:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:32:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:32:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:32:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:32:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:32:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:32:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:32:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:32:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:32:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:32:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:32:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:32:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:32:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:32:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:32:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:32:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:32:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:32:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:32:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:32:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:32:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:32:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:32:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:32:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:32:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:32:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:32:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:32:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:32:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:32:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:32:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:32:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:32:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:32:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:32:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 8
2024-05-08T01:32:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:32:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:32:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:32:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:32:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:32:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:32:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:32:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:32:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:32:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:32:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:32:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:32:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:32:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:32:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:32:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:32:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:32:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:32:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:32:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:32:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:56.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:56.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:56.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:32:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:32:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:32:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:32:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:32:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:32:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:32:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:32:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:32:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:32:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:32:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:32:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:32:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:32:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:32:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:32:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:32:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:32:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:32:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:32:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:32:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:32:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:32:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:32:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:32:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:32:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:32:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:32:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:32:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:32:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:32:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:32:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:32:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:32:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:32:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:32:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:32:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:32:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:32:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:32:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:32:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:32:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:32:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:32:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:32:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:32:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:32:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:32:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:32:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:32:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:32:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:32:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:32:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:32:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:32:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:32:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:32:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:32:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:32:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:32:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:32:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:32:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:32:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:32:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:32:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:32:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:32:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:32:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:32:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:32:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:32:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:32:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:32:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 16
2024-05-08T01:33:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:33:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:33:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:33:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:33:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:33:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:33:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:33:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:33:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:33:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:33:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:33:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:33:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:33:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:33:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:33:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:33:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:33:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:33:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:33:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:33:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:33:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:33:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:33:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:33:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:33:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:33:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:33:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:33:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:33:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:33:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:33:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:33:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:33:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:33:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:33:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:33:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:33:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:33:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:33:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:33:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:33:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:33:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:33:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:33:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:33:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:33:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:33:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:33:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:33:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:33:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:33:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:33:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:33:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:33:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:33:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:33:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:33:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:33:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:33:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:33:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:33:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:33:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:33:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:33:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:33:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:33:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:33:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:33:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:33:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:33:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:33:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:33:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:33:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:33:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:33:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:33:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:33:13.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:33:13.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:33:13.658 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:33:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:33:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:33:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:33:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:33:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:33:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:33:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:33:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:33:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:33:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:33:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:33:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:33:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:33:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:33:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:33:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:33:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:33:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:33:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:33:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:33:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:33:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:33:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:33:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:33:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:33:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:33:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:33:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:33:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:33:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:33:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:33:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:33:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:33:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:33:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:33:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:33:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:33:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:33:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:33:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:33:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:33:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:33:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:33:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:33:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:33:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:33:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:33:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:33:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:33:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:33:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:33:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:33:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:33:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:33:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:33:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:33:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:33:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:33:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:33:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:33:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:33:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:33:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:33:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:33:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:33:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 16
2024-05-08T01:33:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster2 '!=' cluster2 ']'
2024-05-08T01:33:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:33:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:33:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test
2024-05-08T01:33:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:33:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:33:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:33:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:33:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:33:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:33:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:33:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:33:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:33:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:33:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:33:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:33:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:33:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:33:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:33:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:33:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:33:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:33:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:33:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:33:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:33:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:33:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:33:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:33:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:33:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:33:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:33:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:33:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:33:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:33:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:33:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:33:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:33:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test
2024-05-08T01:33:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:33:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:33:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:33:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:33:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test
2024-05-08T01:33:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:33:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:33:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:33:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:33:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:33:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:33:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:33:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:33:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:33:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:33:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:33:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:33:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:33:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:33:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:33:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:33:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:33:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:33:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:33:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:33:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:33:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:33:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:33:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:33:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:33:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:33:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:33:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:33:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:33:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:33:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:33:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:33:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:33:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test
2024-05-08T01:33:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:33:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:33:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:33:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:33:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test
2024-05-08T01:33:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:33:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:33:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:33:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:33:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:33:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:33:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:33:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:33:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:33:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:33:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:33:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:33:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:33:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:33:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:33:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:33:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:33:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:33:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:33:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:33:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:33:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:33:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:33:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:33:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:33:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:33:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:33:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:33:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:33:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:33:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:33:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:33:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:33:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test
2024-05-08T01:33:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:33:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:33:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:33:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:33:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test
2024-05-08T01:33:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:33:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:33:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:33:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:33:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:33:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:33:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:33:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:33:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:33:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:33:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:33:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:33:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:33:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:33:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:33:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:33:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:33:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:33:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:33:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:33:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:33:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:33:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:33:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:33:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:33:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:33:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:33:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:33:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:33:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:33:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:33:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:33:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:33:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test
2024-05-08T01:33:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['pool_to_remove/test']
2024-05-08T01:33:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:33:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:33:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:33:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: snapshot rename'
2024-05-08T01:33:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:33:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:33:30.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:33:30 TEST: snapshot rename
2024-05-08T01:33:30.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:33:30 TEST: snapshot rename
2024-05-08T01:33:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ snap_name=snap_rename
2024-05-08T01:33:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = snapshot ']'
2024-05-08T01:33:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_snapshot cluster2 mirror test2 snap_rename_0
2024-05-08T01:33:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_0
2024-05-08T01:33:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap create mirror/test2@snap_rename_0
2024-05-08T01:33:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Creating snap: 10% complete...
Creating snap: 100% complete...done.
2024-05-08T01:33:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 1 20
2024-05-08T01:33:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 1 - 1
2024-05-08T01:33:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_0 snap_rename_1
2024-05-08T01:33:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_0
2024-05-08T01:33:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_1
2024-05-08T01:33:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_0 mirror/test2@snap_rename_1
2024-05-08T01:33:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 2 - 1
2024-05-08T01:33:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_1 snap_rename_2
2024-05-08T01:33:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_1
2024-05-08T01:33:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_2
2024-05-08T01:33:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_1 mirror/test2@snap_rename_2
2024-05-08T01:33:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 3 - 1
2024-05-08T01:33:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_2 snap_rename_3
2024-05-08T01:33:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_2
2024-05-08T01:33:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_3
2024-05-08T01:33:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_2 mirror/test2@snap_rename_3
2024-05-08T01:33:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 4 - 1
2024-05-08T01:33:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_3 snap_rename_4
2024-05-08T01:33:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_3
2024-05-08T01:33:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_4
2024-05-08T01:33:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_3 mirror/test2@snap_rename_4
2024-05-08T01:33:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 5 - 1
2024-05-08T01:33:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_4 snap_rename_5
2024-05-08T01:33:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_4
2024-05-08T01:33:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_5
2024-05-08T01:33:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_4 mirror/test2@snap_rename_5
2024-05-08T01:33:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 6 - 1
2024-05-08T01:33:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_5 snap_rename_6
2024-05-08T01:33:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_5
2024-05-08T01:33:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_6
2024-05-08T01:33:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_5 mirror/test2@snap_rename_6
2024-05-08T01:33:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 7 - 1
2024-05-08T01:33:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_6 snap_rename_7
2024-05-08T01:33:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_6
2024-05-08T01:33:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_7
2024-05-08T01:33:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_6 mirror/test2@snap_rename_7
2024-05-08T01:33:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 8 - 1
2024-05-08T01:33:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_7 snap_rename_8
2024-05-08T01:33:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_7
2024-05-08T01:33:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_8
2024-05-08T01:33:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_7 mirror/test2@snap_rename_8
2024-05-08T01:33:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 9 - 1
2024-05-08T01:33:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_8 snap_rename_9
2024-05-08T01:33:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_8
2024-05-08T01:33:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_9
2024-05-08T01:33:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_8 mirror/test2@snap_rename_9
2024-05-08T01:33:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 10 - 1
2024-05-08T01:33:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_9 snap_rename_10
2024-05-08T01:33:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_9
2024-05-08T01:33:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_10
2024-05-08T01:33:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_9 mirror/test2@snap_rename_10
2024-05-08T01:33:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 11 - 1
2024-05-08T01:33:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_10 snap_rename_11
2024-05-08T01:33:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_10
2024-05-08T01:33:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_11
2024-05-08T01:33:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_10 mirror/test2@snap_rename_11
2024-05-08T01:33:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 12 - 1
2024-05-08T01:33:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_11 snap_rename_12
2024-05-08T01:33:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_11
2024-05-08T01:33:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_12
2024-05-08T01:33:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_11 mirror/test2@snap_rename_12
2024-05-08T01:33:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 13 - 1
2024-05-08T01:33:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_12 snap_rename_13
2024-05-08T01:33:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:32.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:32.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_12
2024-05-08T01:33:32.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_13
2024-05-08T01:33:32.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_12 mirror/test2@snap_rename_13
2024-05-08T01:33:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 14 - 1
2024-05-08T01:33:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_13 snap_rename_14
2024-05-08T01:33:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_13
2024-05-08T01:33:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_14
2024-05-08T01:33:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_13 mirror/test2@snap_rename_14
2024-05-08T01:33:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 15 - 1
2024-05-08T01:33:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_14 snap_rename_15
2024-05-08T01:33:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_14
2024-05-08T01:33:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_15
2024-05-08T01:33:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_14 mirror/test2@snap_rename_15
2024-05-08T01:33:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 16 - 1
2024-05-08T01:33:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_15 snap_rename_16
2024-05-08T01:33:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_15
2024-05-08T01:33:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_16
2024-05-08T01:33:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_15 mirror/test2@snap_rename_16
2024-05-08T01:33:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 17 - 1
2024-05-08T01:33:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_16 snap_rename_17
2024-05-08T01:33:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_16
2024-05-08T01:33:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_17
2024-05-08T01:33:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_16 mirror/test2@snap_rename_17
2024-05-08T01:33:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 18 - 1
2024-05-08T01:33:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_17 snap_rename_18
2024-05-08T01:33:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_17
2024-05-08T01:33:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_18
2024-05-08T01:33:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_17 mirror/test2@snap_rename_18
2024-05-08T01:33:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 19 - 1
2024-05-08T01:33:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_18 snap_rename_19
2024-05-08T01:33:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_18
2024-05-08T01:33:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_19
2024-05-08T01:33:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_18 mirror/test2@snap_rename_19
2024-05-08T01:33:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in `seq 1 20`
2024-05-08T01:33:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ expr 20 - 1
2024-05-08T01:33:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_19 snap_rename_20
2024-05-08T01:33:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap_rename_19
2024-05-08T01:33:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local new_snap=snap_rename_20
2024-05-08T01:33:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_19 mirror/test2@snap_rename_20
2024-05-08T01:33:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = snapshot ']'
2024-05-08T01:33:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_snap_present cluster1 mirror test2 snap_rename_20
2024-05-08T01:33:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:33:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap_name=snap_rename_20
2024-05-08T01:33:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:33:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:33:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:33:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20
2024-05-08T01:33:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:error setting snapshot context: (2) No such file or directory
2024-05-08T01:33:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ continue
2024-05-08T01:33:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:33:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:33:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20
2024-05-08T01:33:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:error setting snapshot context: (2) No such file or directory
2024-05-08T01:33:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ continue
2024-05-08T01:33:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:33:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 4
2024-05-08T01:33:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20
2024-05-08T01:33:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:error setting snapshot context: (2) No such file or directory
2024-05-08T01:33:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ continue
2024-05-08T01:33:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:33:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 8
2024-05-08T01:33:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20
2024-05-08T01:33:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:error setting snapshot context: (2) No such file or directory
2024-05-08T01:33:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ continue
2024-05-08T01:33:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32
2024-05-08T01:33:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 8
2024-05-08T01:33:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20
2024-05-08T01:33:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:rbd image 'test2':
2024-05-08T01:33:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	size 128 MiB in 32 objects
2024-05-08T01:33:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	order 22 (4 MiB objects)
2024-05-08T01:33:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	snapshot_count: 3
2024-05-08T01:33:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	id: 10ded2b61589
2024-05-08T01:33:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	block_name_prefix: rbd_data.10ded2b61589
2024-05-08T01:33:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	format: 2
2024-05-08T01:33:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	features: layering, exclusive-lock, journaling
2024-05-08T01:33:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	op_features:
2024-05-08T01:33:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	flags:
2024-05-08T01:33:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	create_timestamp: Wed May  8 01:29:13 2024
2024-05-08T01:33:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	access_timestamp: Wed May  8 01:29:13 2024
2024-05-08T01:33:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	modify_timestamp: Wed May  8 01:29:13 2024
2024-05-08T01:33:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	protected: False
2024-05-08T01:33:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	journal: 10ded2b61589
2024-05-08T01:33:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring state: enabled
2024-05-08T01:33:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring mode: journal
2024-05-08T01:33:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring global id: 913b5117-3b0e-441b-89b0-b86584f55335
2024-05-08T01:33:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring primary: false
2024-05-08T01:33:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:33:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ unprotect_snapshot cluster2 mirror test4 snap1
2024-05-08T01:33:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:33:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap1
2024-05-08T01:33:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test4@snap1
2024-05-08T01:33:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ unprotect_snapshot cluster2 mirror test4 snap2
2024-05-08T01:33:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:33:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local snap=snap2
2024-05-08T01:33:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test4@snap2
2024-05-08T01:33:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in ${image2} ${image4}
2024-05-08T01:33:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster2 mirror test2
2024-05-08T01:33:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:33:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:33:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster2 mirror test2
2024-05-08T01:33:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test2
2024-05-08T01:33:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 snap purge mirror/test2
2024-05-08T01:33:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing all snapshots: 33% complete...
Removing all snapshots: 66% complete...
Removing all snapshots: 100% complete...
Removing all snapshots: 100% complete...done.
2024-05-08T01:33:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rm mirror/test2
2024-05-08T01:33:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:33:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:33:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for i in ${image2} ${image4}
2024-05-08T01:33:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster2 mirror test4
2024-05-08T01:33:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:33:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:33:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:33:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster2 mirror test4
2024-05-08T01:33:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:33:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:33:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test4
2024-05-08T01:33:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 snap purge mirror/test4
2024-05-08T01:34:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing all snapshots: 50% complete...
Removing all snapshots: 100% complete...
Removing all snapshots: 100% complete...done.
2024-05-08T01:34:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rm mirror/test4
2024-05-08T01:34:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:34:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: disable mirror while daemon is stopped'
2024-05-08T01:34:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:34:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:34:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:34:01 TEST: disable mirror while daemon is stopped
2024-05-08T01:34:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:34:01 TEST: disable mirror while daemon is stopped
2024-05-08T01:34:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirrors cluster1
2024-05-08T01:34:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:34:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:34:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster1:0 ''
2024-05-08T01:34:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:34:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:34:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:34:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster1:0
2024-05-08T01:34:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1:0
2024-05-08T01:34:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:34:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:34:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:34:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster1
2024-05-08T01:34:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster1
2024-05-08T01:34:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:34:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:34:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file'
2024-05-08T01:34:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid
2024-05-08T01:34:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid
2024-05-08T01:34:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=31313
2024-05-08T01:34:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 31313 ']'
2024-05-08T01:34:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 31313
2024-05-08T01:34:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:34:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:34:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:34:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=31313 '$2 == pid {print; exit 1}'
2024-05-08T01:34:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:ubuntu     31313  1.0  0.2 553600 68644 ?        Ssl  01:16   0:11 rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:34:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:34:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:34:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:34:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=31313 '$2 == pid {print; exit 1}'
2024-05-08T01:34:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:34:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:34:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=31313 '$2 == pid {print; exit 1}'
2024-05-08T01:34:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:34:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:34:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:34:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:34:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:34:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:34:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:34:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:04.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster2
2024-05-08T01:34:04.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:34:04.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:04.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:04.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:34:04.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:34:04.799 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:34:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:34:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:34:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster1:0
2024-05-08T01:34:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:34:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:34:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:34:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file'
2024-05-08T01:34:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid
2024-05-08T01:34:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid
2024-05-08T01:34:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster1:1 ''
2024-05-08T01:34:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:34:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:34:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:34:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster1:1
2024-05-08T01:34:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1:1
2024-05-08T01:34:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:34:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:34:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:34:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:34:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:34:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster1
2024-05-08T01:34:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster1
2024-05-08T01:34:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:34:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:34:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file'
2024-05-08T01:34:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid
2024-05-08T01:34:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid
2024-05-08T01:34:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=31341
2024-05-08T01:34:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 31341 ']'
2024-05-08T01:34:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 31341
2024-05-08T01:34:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:34:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:34:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:34:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=31341 '$2 == pid {print; exit 1}'
2024-05-08T01:34:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:34:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:34:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=31341 '$2 == pid {print; exit 1}'
2024-05-08T01:34:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:34:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:34:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:34:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:34:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:34:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:34:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:34:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:34:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:34:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:34:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:34:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:34:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:34:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster2
2024-05-08T01:34:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:34:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:34:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:34:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:34:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:34:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:34:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:34:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:34:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:34:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:34:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:34:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:34:05.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster1:1
2024-05-08T01:34:05.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:34:05.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:34:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:34:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:34:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:34:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:34:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:34:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file'
2024-05-08T01:34:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid
2024-05-08T01:34:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid
2024-05-08T01:34:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster1:2 ''
2024-05-08T01:34:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:34:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:34:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:34:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster1:2
2024-05-08T01:34:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1:2
2024-05-08T01:34:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:34:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:34:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:34:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:34:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:34:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster1
2024-05-08T01:34:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster1
2024-05-08T01:34:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:34:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:34:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file'
2024-05-08T01:34:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid
2024-05-08T01:34:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid
2024-05-08T01:34:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=31369
2024-05-08T01:34:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 31369 ']'
2024-05-08T01:34:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 31369
2024-05-08T01:34:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:34:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:34:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:34:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=31369 '$2 == pid {print; exit 1}'
2024-05-08T01:34:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:34:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:34:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=31369 '$2 == pid {print; exit 1}'
2024-05-08T01:34:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:34:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:34:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:34:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:34:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:34:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:34:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:34:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:34:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:34:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:34:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:34:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:34:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:34:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster2
2024-05-08T01:34:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:34:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:34:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:34:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:34:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:34:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:34:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:34:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:34:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:34:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:34:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:34:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:34:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster1:2
2024-05-08T01:34:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:34:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:34:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:34:07.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:07.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:07.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:07.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:34:07.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:34:07.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:07.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:07.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:34:07.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:34:07.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file'
2024-05-08T01:34:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid
2024-05-08T01:34:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid
2024-05-08T01:34:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster1:3 ''
2024-05-08T01:34:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:34:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:34:07.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:07.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:34:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster1:3
2024-05-08T01:34:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1:3
2024-05-08T01:34:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:34:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:34:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:34:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:34:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:34:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster1
2024-05-08T01:34:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster1
2024-05-08T01:34:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:34:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:34:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file'
2024-05-08T01:34:07.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid
2024-05-08T01:34:07.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid
2024-05-08T01:34:07.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=31439
2024-05-08T01:34:07.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 31439 ']'
2024-05-08T01:34:07.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 31439
2024-05-08T01:34:07.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:34:07.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:34:08.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:34:08.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=31439 '$2 == pid {print; exit 1}'
2024-05-08T01:34:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:34:08.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:34:08.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=31439 '$2 == pid {print; exit 1}'
2024-05-08T01:34:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:34:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:34:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:34:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:34:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:34:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:34:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:34:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:34:08.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:34:08.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:34:08.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:34:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:34:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:34:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster2
2024-05-08T01:34:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:34:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:34:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:34:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:34:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:34:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:34:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:34:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:34:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:34:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:34:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:34:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:34:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster1:3
2024-05-08T01:34:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:34:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:34:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:34:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:34:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:34:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:34:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:34:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file'
2024-05-08T01:34:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid
2024-05-08T01:34:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid
2024-05-08T01:34:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirrors cluster2
2024-05-08T01:34:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:34:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:34:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster2:0 ''
2024-05-08T01:34:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:34:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:34:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:34:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster2:0
2024-05-08T01:34:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2:0
2024-05-08T01:34:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:34:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:34:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:34:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:34:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:34:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster2
2024-05-08T01:34:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster2
2024-05-08T01:34:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:34:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:34:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file'
2024-05-08T01:34:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid
2024-05-08T01:34:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid
2024-05-08T01:34:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=35202
2024-05-08T01:34:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 35202 ']'
2024-05-08T01:34:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 35202
2024-05-08T01:34:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:34:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:34:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:34:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=35202 '$2 == pid {print; exit 1}'
2024-05-08T01:34:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:34:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:34:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=35202 '$2 == pid {print; exit 1}'
2024-05-08T01:34:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster1
2024-05-08T01:34:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:34:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:34:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:34:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:34:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:34:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:34:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:34:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:34:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:34:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:34:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:34:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:34:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:34:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:34:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:34:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:09.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:34:09.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:34:09.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:34:09.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:09.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:09.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:34:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:34:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:34:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster2:0
2024-05-08T01:34:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:34:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:34:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:34:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:34:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:34:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:34:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:34:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file'
2024-05-08T01:34:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid
2024-05-08T01:34:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid
2024-05-08T01:34:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster2:1 ''
2024-05-08T01:34:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:34:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:34:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:34:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster2:1
2024-05-08T01:34:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2:1
2024-05-08T01:34:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:34:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:1
2024-05-08T01:34:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:34:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:34:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:34:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster2:1 ']'
2024-05-08T01:34:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster2
2024-05-08T01:34:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster2
2024-05-08T01:34:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:34:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:34:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file'
2024-05-08T01:34:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid
2024-05-08T01:34:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid
2024-05-08T01:34:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=35230
2024-05-08T01:34:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 35230 ']'
2024-05-08T01:34:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 35230
2024-05-08T01:34:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:34:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:34:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:34:10.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=35230 '$2 == pid {print; exit 1}'
2024-05-08T01:34:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:34:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:34:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=35230 '$2 == pid {print; exit 1}'
2024-05-08T01:34:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster1
2024-05-08T01:34:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:34:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:34:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:34:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:34:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:34:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:34:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:34:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:34:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:34:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:34:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:34:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:34:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:34:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:34:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:34:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:34:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:34:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:34:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:34:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:34:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:34:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:34:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:34:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:34:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:34:10.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:34:10.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:34:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster2:1
2024-05-08T01:34:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:1
2024-05-08T01:34:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:34:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:34:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:34:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:34:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:34:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:34:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:34:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:34:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:34:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file'
2024-05-08T01:34:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid
2024-05-08T01:34:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid
2024-05-08T01:34:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster2:2 ''
2024-05-08T01:34:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:34:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:34:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:34:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster2:2
2024-05-08T01:34:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2:2
2024-05-08T01:34:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:34:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:2
2024-05-08T01:34:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:34:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:34:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:34:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster2:2 ']'
2024-05-08T01:34:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster2
2024-05-08T01:34:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster2
2024-05-08T01:34:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:34:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:34:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.2 'pid file'
2024-05-08T01:34:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid
2024-05-08T01:34:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid
2024-05-08T01:34:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=35274
2024-05-08T01:34:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 35274 ']'
2024-05-08T01:34:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 35274
2024-05-08T01:34:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:34:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:34:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:34:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=35274 '$2 == pid {print; exit 1}'
2024-05-08T01:34:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:34:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:34:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=35274 '$2 == pid {print; exit 1}'
2024-05-08T01:34:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster1
2024-05-08T01:34:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:34:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:34:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:34:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:34:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:34:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:34:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:34:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:34:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:34:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:34:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:34:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:34:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:34:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:34:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:34:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:34:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:34:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:34:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:34:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:34:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:34:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:34:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:34:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:34:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:34:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:34:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:34:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster2:2
2024-05-08T01:34:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:2
2024-05-08T01:34:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:34:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:34:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:34:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:34:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:34:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:34:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:34:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:34:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:34:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'pid file'
2024-05-08T01:34:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid
2024-05-08T01:34:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid
2024-05-08T01:34:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster2:3 ''
2024-05-08T01:34:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:34:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:34:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:34:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster2:3
2024-05-08T01:34:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2:3
2024-05-08T01:34:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:34:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:3
2024-05-08T01:34:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:34:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:34:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:34:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster2:3 ']'
2024-05-08T01:34:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster2
2024-05-08T01:34:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster2
2024-05-08T01:34:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:34:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:34:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.3 'pid file'
2024-05-08T01:34:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid
2024-05-08T01:34:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid
2024-05-08T01:34:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=35334
2024-05-08T01:34:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 35334 ']'
2024-05-08T01:34:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 35334
2024-05-08T01:34:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:34:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:34:12.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:34:12.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=35334 '$2 == pid {print; exit 1}'
2024-05-08T01:34:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:34:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:34:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=35334 '$2 == pid {print; exit 1}'
2024-05-08T01:34:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster1
2024-05-08T01:34:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:34:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:34:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:34:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:34:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:34:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:34:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:34:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:34:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:34:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:34:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:34:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:34:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:34:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:34:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:34:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:34:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:34:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:34:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:34:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:34:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:34:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:34:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:34:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:34:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:34:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:34:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:34:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster2:3
2024-05-08T01:34:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:3
2024-05-08T01:34:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:34:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:34:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:34:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:34:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:34:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:34:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:34:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:34:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:34:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'pid file'
2024-05-08T01:34:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid
2024-05-08T01:34:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid
2024-05-08T01:34:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:34:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_pool_mirror_mode cluster2 mirror image
2024-05-08T01:34:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=image
2024-05-08T01:34:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image
2024-05-08T01:34:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:note: changing mirroring mode from pool to image
2024-05-08T01:34:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ disable_mirror cluster2 mirror test
2024-05-08T01:34:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/test
2024-05-08T01:34:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Mirroring disabled
2024-05-08T01:34:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -z '' ']'
2024-05-08T01:34:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test present
2024-05-08T01:34:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:34:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:34:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:34:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:34:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:34:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:34:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:34:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=10deccb08620
2024-05-08T01:34:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 10deccb08620
2024-05-08T01:34:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z '' -o '' = 10deccb08620
2024-05-08T01:34:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:34:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = present
2024-05-08T01:34:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirrors cluster1
2024-05-08T01:34:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:34:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster1:0
2024-05-08T01:34:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:34:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:34:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:34:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:34:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:34:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:34:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:34:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:34:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:34:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:34:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:34:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster1:1
2024-05-08T01:34:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:34:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:34:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:34:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:1
2024-05-08T01:34:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:34:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:34:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:34:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:34:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster1:1 ']'
2024-05-08T01:34:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:34:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:34:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:34:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:34:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster1:2
2024-05-08T01:34:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:34:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:34:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:34:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:2
2024-05-08T01:34:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:34:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:34:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:34:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:34:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster1:2 ']'
2024-05-08T01:34:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:34:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:34:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:34:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:34:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster1:3
2024-05-08T01:34:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:34:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:34:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:34:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:3
2024-05-08T01:34:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:34:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:34:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:34:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:34:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster1:3 ']'
2024-05-08T01:34:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:34:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:34:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:34:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:34:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test deleted
2024-05-08T01:34:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=deleted
2024-05-08T01:34:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:34:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:34:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:34:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:34:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:34:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=10deccb08620
2024-05-08T01:34:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:34:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:34:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test deleted 10deccb08620
2024-05-08T01:34:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:34:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=10deccb08620
2024-05-08T01:34:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:34:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:34:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:34:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:34:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:34:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=10deccb08620
2024-05-08T01:34:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 10deccb08620
2024-05-08T01:34:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z 10deccb08620 -o 10deccb08620 = 10deccb08620
2024-05-08T01:34:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:34:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = present
2024-05-08T01:34:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:34:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:34:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test deleted 10deccb08620
2024-05-08T01:34:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:34:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=10deccb08620
2024-05-08T01:34:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:34:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:34:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:34:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:34:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:34:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=10deccb08620
2024-05-08T01:34:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 10deccb08620
2024-05-08T01:34:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z 10deccb08620 -o 10deccb08620 = 10deccb08620
2024-05-08T01:34:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:34:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = present
2024-05-08T01:34:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:34:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:34:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test deleted 10deccb08620
2024-05-08T01:34:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:34:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=10deccb08620
2024-05-08T01:34:16.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:34:16.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:34:16.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:34:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:34:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:34:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=10deccb08620
2024-05-08T01:34:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 10deccb08620
2024-05-08T01:34:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z 10deccb08620 -o 10deccb08620 = 10deccb08620
2024-05-08T01:34:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:34:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = present
2024-05-08T01:34:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:34:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 4
2024-05-08T01:34:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test deleted 10deccb08620
2024-05-08T01:34:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:34:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=10deccb08620
2024-05-08T01:34:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:34:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:34:20.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:34:20.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:20.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:34:20.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:20.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:34:20.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test: (2) No such file or directory
2024-05-08T01:34:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:34:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = deleted
2024-05-08T01:34:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = snapshot ']'
2024-05-08T01:34:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_pool_mirror_mode cluster2 mirror pool
2024-05-08T01:34:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=pool
2024-05-08T01:34:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror pool
2024-05-08T01:34:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:note: changing mirroring mode from image to pool
2024-05-08T01:34:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ enable_journaling cluster2 mirror test
2024-05-08T01:34:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 feature enable mirror/test journaling
2024-05-08T01:34:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test present
2024-05-08T01:34:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=present
2024-05-08T01:34:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:34:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:34:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:34:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:34:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:34:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test: (2) No such file or directory
2024-05-08T01:34:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=
2024-05-08T01:34:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:34:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:34:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test present ''
2024-05-08T01:34:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:34:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:34:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:34:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:34:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:34:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:34:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:34:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test: (2) No such file or directory
2024-05-08T01:34:21.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:34:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:34:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:34:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:34:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test present ''
2024-05-08T01:34:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:34:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:34:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:34:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:34:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:34:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:34:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:34:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test: (2) No such file or directory
2024-05-08T01:34:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:34:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:34:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:34:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:34:24.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test present ''
2024-05-08T01:34:24.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:24.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:24.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:24.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:34:24.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:34:24.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:34:24.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:34:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:34:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:34:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:34:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test: (2) No such file or directory
2024-05-08T01:34:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:34:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:34:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:34:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 4
2024-05-08T01:34:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test present ''
2024-05-08T01:34:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:34:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:34:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:34:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:34:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:34:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:34:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:34:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=16f0e3b9669d
2024-05-08T01:34:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 16f0e3b9669d
2024-05-08T01:34:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z '' -o '' = 16f0e3b9669d
2024-05-08T01:34:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:34:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = present
2024-05-08T01:34:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:34:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:34:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:34:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:34:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:34:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:34:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:34:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:34:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:34:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:34:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:34:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:34:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:34:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:34:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:34:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:34:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:34:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:34:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:34:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:34:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:34:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:34:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:34:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:34:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:34:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:34:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:34:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:34:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:34:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:34:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test
2024-05-08T01:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:34:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:34:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:34:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:34:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:34:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:34:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:34:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:34:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test
2024-05-08T01:34:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:34:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test
2024-05-08T01:34:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:34:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:34:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:34:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:34:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:34:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:34:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:34:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:34:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:34:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:34:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:34:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:34:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:34:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:34:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:34:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:34:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:34:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:34:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:34:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test
2024-05-08T01:34:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:34:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test
2024-05-08T01:34:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:34:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:34:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:34:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:34:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:34:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:34:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:34:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:34:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:34:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:34:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:34:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:34:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:34:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:34:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:34:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:34:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:34:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:34:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:34:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test
2024-05-08T01:34:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/test']
2024-05-08T01:34:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 1
2024-05-08T01:34:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:34:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:34:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:34:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:34:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:34:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:34:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:34:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:34:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:34:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:34:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:34:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:34:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:34:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:34:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:34:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:34:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:34:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:34:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:34:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:34:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:34:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: non-default namespace image mirroring'
2024-05-08T01:34:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:34:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:34:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:34:31 TEST: non-default namespace image mirroring
2024-05-08T01:34:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:34:31 TEST: non-default namespace image mirroring
2024-05-08T01:34:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog ' - replay'
2024-05-08T01:34:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:34:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:34:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:34:31 - replay
2024-05-08T01:34:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:34:31 - replay
2024-05-08T01:34:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image_and_enable_mirror cluster2 mirror/ns1 test journal
2024-05-08T01:34:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:34:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n journal ']'
2024-05-08T01:34:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image cluster2 mirror/ns1 test
2024-05-08T01:34:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=128
2024-05-08T01:34:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:34:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/ns1/test
2024-05-08T01:34:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ is_pool_mirror_mode_image mirror/ns1
2024-05-08T01:34:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' pool = image ']'
2024-05-08T01:34:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ case "${pool}" in
2024-05-08T01:34:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:34:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image_and_enable_mirror cluster2 mirror/ns2 test journal
2024-05-08T01:34:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:34:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n journal ']'
2024-05-08T01:34:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image cluster2 mirror/ns2 test
2024-05-08T01:34:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=128
2024-05-08T01:34:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:34:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/ns2/test
2024-05-08T01:34:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ is_pool_mirror_mode_image mirror/ns2
2024-05-08T01:34:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' pool = image ']'
2024-05-08T01:34:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ case "${pool}" in
2024-05-08T01:34:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ enable_mirror cluster2 mirror/ns2 test journal
2024-05-08T01:34:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:34:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/ns2/test journal
2024-05-08T01:34:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Mirroring enabled
2024-05-08T01:34:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 info mirror/ns2/test
2024-05-08T01:34:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:rbd image 'test':
2024-05-08T01:34:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	size 128 MiB in 32 objects
2024-05-08T01:34:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	order 22 (4 MiB objects)
2024-05-08T01:34:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	snapshot_count: 0
2024-05-08T01:34:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	id: 18267f9eb705
2024-05-08T01:34:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	block_name_prefix: rbd_data.18267f9eb705
2024-05-08T01:34:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	format: 2
2024-05-08T01:34:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	features: layering, exclusive-lock, journaling
2024-05-08T01:34:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	op_features:
2024-05-08T01:34:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	flags:
2024-05-08T01:34:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	create_timestamp: Wed May  8 01:34:31 2024
2024-05-08T01:34:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	access_timestamp: Wed May  8 01:34:31 2024
2024-05-08T01:34:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	modify_timestamp: Wed May  8 01:34:31 2024
2024-05-08T01:34:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	journal: 18267f9eb705
2024-05-08T01:34:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring state: enabled
2024-05-08T01:34:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring mode: journal
2024-05-08T01:34:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring global id: 9323a208-7ab1-450b-bc8b-d7543fc4ad3a
2024-05-08T01:34:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:	mirroring primary: true
2024-05-08T01:34:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror/ns1 test
2024-05-08T01:34:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror/ns1 test started
2024-05-08T01:34:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:34:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:34:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:34:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror/ns1 test started
2024-05-08T01:34:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:34:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:34:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:34:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/ns1/test
2024-05-08T01:34:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:34:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:34:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:34:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:34:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:34:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:34:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:34:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test
2024-05-08T01:34:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:34:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:34:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:34:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:34:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:34:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:34:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:34:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:34:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:34:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:34:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test
2024-05-08T01:34:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/ns1/test']
2024-05-08T01:34:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test
2024-05-08T01:34:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:34:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:34:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:34:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:34:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:34:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:34:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:34:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:34:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:34:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:34:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:34:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:34:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:34:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:34:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:34:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:34:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:34:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:34:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:34:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test
2024-05-08T01:34:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/ns1/test']
2024-05-08T01:34:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/ns1/test
2024-05-08T01:34:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:34:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:34:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:34:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:34:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:34:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:34:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:34:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:34:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:34:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:34:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:34:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:34:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:34:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:34:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:34:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:34:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:34:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:34:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:34:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns1/test
2024-05-08T01:34:32.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/ns1/test']
2024-05-08T01:34:32.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:32.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/ns1/test
2024-05-08T01:34:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:34:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:34:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:34:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:34:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:34:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:34:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:34:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:34:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:34:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:34:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:34:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:34:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:34:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:34:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:34:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:34:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:34:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:34:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:34:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns1/test
2024-05-08T01:34:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/ns1/test']
2024-05-08T01:34:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 1
2024-05-08T01:34:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:34:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:34:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test
2024-05-08T01:34:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:34:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:34:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:34:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:34:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:34:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:34:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test
2024-05-08T01:34:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/ns1/test']
2024-05-08T01:34:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test
2024-05-08T01:34:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:34:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:34:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:34:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:34:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:34:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:34:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:34:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:34:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:34:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:34:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:34:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:34:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:34:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:34:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:34:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:34:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:34:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:34:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:34:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test
2024-05-08T01:34:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/ns1/test']
2024-05-08T01:34:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/ns1/test
2024-05-08T01:34:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:34:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:34:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:34:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:34:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:34:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:34:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:34:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:34:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:34:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:34:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:34:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:34:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:34:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:34:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:34:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:34:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:34:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:34:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:34:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns1/test
2024-05-08T01:34:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/ns1/test']
2024-05-08T01:34:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/ns1/test
2024-05-08T01:34:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:34:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:34:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:34:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:34:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:34:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:34:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:34:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:34:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:34:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:34:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:34:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:34:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:34:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:34:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:34:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:34:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:34:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:34:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:34:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns1/test
2024-05-08T01:34:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/ns1/test']
2024-05-08T01:34:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 2
2024-05-08T01:34:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:34:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:34:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test
2024-05-08T01:34:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:34:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:34:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:34:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:34:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:34:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:34:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:34:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:34:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:34:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:34:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test
2024-05-08T01:34:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:34:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:34:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:34:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:34:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:34:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:34:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:34:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror/ns2 test
2024-05-08T01:34:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror/ns2 test started
2024-05-08T01:34:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:34:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:34:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:34:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror/ns2 test started
2024-05-08T01:34:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:34:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:34:37.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:34:37.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/ns2/test
2024-05-08T01:34:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:34:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:34:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:34:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:34:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:34:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:34:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:34:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns2/test
2024-05-08T01:34:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:34:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:34:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:34:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:34:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:34:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:34:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:34:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:34:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:34:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:34:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns2/test
2024-05-08T01:34:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:34:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:34:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:34:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:34:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:34:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:34:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:34:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror/ns1 test 100
2024-05-08T01:34:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:34:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:34:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:34:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/ns1/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:34:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:34:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:34:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2083.39   bytes/sec: 8.1 MiB/s
2024-05-08T01:34:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror/ns2 test 100
2024-05-08T01:34:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:34:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:34:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:34:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/ns2/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:34:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:34:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:34:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 2500.07   bytes/sec: 9.8 MiB/s
2024-05-08T01:34:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns1 test
2024-05-08T01:34:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:34:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:34:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror/ns1 test
2024-05-08T01:34:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:34:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:34:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:34:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:34:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:34:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:34:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror/ns1 test
2024-05-08T01:34:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:34:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:34:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/ns1/test'
2024-05-08T01:34:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test
2024-05-08T01:34:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:34:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:34:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:34:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:34:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:34:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:34:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test
2024-05-08T01:34:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:34:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:34:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:34:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:34:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:34:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:34:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:34:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:34:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:34:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:34:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:34:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:34:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:34:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:34:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:34:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:34:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:34:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test
2024-05-08T01:34:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror/ns1 test
2024-05-08T01:34:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror/ns1 test ''
2024-05-08T01:34:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:34:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror/ns1-test.status
2024-05-08T01:34:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror/ns1-test.status
2024-05-08T01:34:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:34:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.status
2024-05-08T01:34:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.status
2024-05-08T01:34:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test
2024-05-08T01:34:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:34:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:34:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:34:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:34:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:34:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.status
2024-05-08T01:34:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:34:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test
2024-05-08T01:34:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror/ns1 test '..*'
2024-05-08T01:34:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:34:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror/ns1-test.status
2024-05-08T01:34:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror/ns1-test.status
2024-05-08T01:34:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:34:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.status
2024-05-08T01:34:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test
2024-05-08T01:34:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.status
2024-05-08T01:34:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:34:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:34:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:34:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:34:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected]
2024-05-08T01:34:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.status
2024-05-08T01:34:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:34:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:34:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' '!=' ''
2024-05-08T01:34:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:34:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' = ''
2024-05-08T01:34:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:34:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:34:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:34:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:34:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=2
2024-05-08T01:34:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:34:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:34:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:34:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=1
2024-05-08T01:34:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:34:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:34:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:34:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=101
2024-05-08T01:34:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:34:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:34:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:34:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=3
2024-05-08T01:34:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 2 = 1 -a 101 -le 3
2024-05-08T01:34:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:34:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:34:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:34:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror/ns1 test
2024-05-08T01:34:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:34:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:34:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/ns1/test'
2024-05-08T01:34:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test
2024-05-08T01:34:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:34:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:34:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:34:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:34:38.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:34:38.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:34:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test
2024-05-08T01:34:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:34:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:34:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:34:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:34:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:34:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:34:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:34:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:34:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:34:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:34:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:34:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:34:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:34:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:34:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:34:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:34:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:34:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test
2024-05-08T01:34:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror/ns1 test
2024-05-08T01:34:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror/ns1 test ''
2024-05-08T01:34:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:34:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror/ns1-test.status
2024-05-08T01:34:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror/ns1-test.status
2024-05-08T01:34:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:34:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.status
2024-05-08T01:34:38.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test
2024-05-08T01:34:38.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.status
2024-05-08T01:34:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:34:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:34:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:34:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:34:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=25], [object_number=0, tag_tid=2, entry_tid=24], [object_number=3, tag_tid=2, entry_tid=23], [object_number=2, tag_tid=2, entry_tid=22]]], state=connected]
2024-05-08T01:34:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.status
2024-05-08T01:34:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:34:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test
2024-05-08T01:34:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror/ns1 test '..*'
2024-05-08T01:34:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:34:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror/ns1-test.status
2024-05-08T01:34:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror/ns1-test.status
2024-05-08T01:34:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:34:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.status
2024-05-08T01:34:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test
2024-05-08T01:34:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.status
2024-05-08T01:34:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:34:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:34:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:34:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:34:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=25], [object_number=0, tag_tid=2, entry_tid=24], [object_number=3, tag_tid=2, entry_tid=23], [object_number=2, tag_tid=2, entry_tid=22]]], state=connected]
2024-05-08T01:34:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.status
2024-05-08T01:34:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=25'
2024-05-08T01:34:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=25'
2024-05-08T01:34:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=25' '!=' 'object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:34:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:34:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=25' = 'object_number=3, tag_tid=1, entry_tid=3'
2024-05-08T01:34:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=1, tag_tid=2, entry_tid=25'
2024-05-08T01:34:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:34:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:34:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:34:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=2
2024-05-08T01:34:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=25'
2024-05-08T01:34:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:34:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:34:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=2
2024-05-08T01:34:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:34:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:34:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:34:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=101
2024-05-08T01:34:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=25'
2024-05-08T01:34:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:34:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:34:38.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=25
2024-05-08T01:34:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 2 = 2 -a 101 -le 25
2024-05-08T01:34:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:34:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:34:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:34:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror/ns1 test
2024-05-08T01:34:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:34:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:34:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/ns1/test'
2024-05-08T01:34:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test
2024-05-08T01:34:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:34:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:34:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:34:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:34:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:34:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:34:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test
2024-05-08T01:34:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:34:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:34:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:34:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:34:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:34:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:34:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:34:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:34:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:34:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:34:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:34:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:34:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:34:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:34:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:34:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:34:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:34:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test
2024-05-08T01:34:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror/ns1 test
2024-05-08T01:34:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror/ns1 test ''
2024-05-08T01:34:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:34:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror/ns1-test.status
2024-05-08T01:34:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:34:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror/ns1-test.status
2024-05-08T01:34:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.status
2024-05-08T01:34:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test
2024-05-08T01:34:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.status
2024-05-08T01:34:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:34:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:34:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:34:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:34:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:34:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.status
2024-05-08T01:34:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:34:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test
2024-05-08T01:34:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror/ns1 test '..*'
2024-05-08T01:34:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:34:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror/ns1-test.status
2024-05-08T01:34:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror/ns1-test.status
2024-05-08T01:34:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:34:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.status
2024-05-08T01:34:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test
2024-05-08T01:34:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.status
2024-05-08T01:34:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:34:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:34:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:34:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:34:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:34:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.status
2024-05-08T01:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns2 test
2024-05-08T01:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror/ns2 test
2024-05-08T01:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:34:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:34:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:34:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:34:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:34:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror/ns2 test
2024-05-08T01:34:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:34:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test ']'
2024-05-08T01:34:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/ns2/test'
2024-05-08T01:34:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns2/test
2024-05-08T01:34:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:34:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:34:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:34:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:34:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:34:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:34:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns2/test
2024-05-08T01:34:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:34:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:34:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:34:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:34:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:34:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:34:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:34:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:34:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:34:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:34:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:34:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:34:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:34:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:34:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:34:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:34:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:34:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns2/test
2024-05-08T01:34:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror/ns2 test
2024-05-08T01:34:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns2
2024-05-08T01:34:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror/ns2 test ''
2024-05-08T01:34:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns2
2024-05-08T01:34:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:34:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror/ns2-test.status
2024-05-08T01:34:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror/ns2-test.status
2024-05-08T01:34:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:34:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror_ns2-test.status
2024-05-08T01:34:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns2/test
2024-05-08T01:34:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror_ns2-test.status
2024-05-08T01:34:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:34:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:34:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:34:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:34:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:34:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror_ns2-test.status
2024-05-08T01:34:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:34:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror/ns2 test
2024-05-08T01:34:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns2
2024-05-08T01:34:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror/ns2 test '..*'
2024-05-08T01:34:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns2
2024-05-08T01:34:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:34:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror/ns2-test.status
2024-05-08T01:34:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror/ns2-test.status
2024-05-08T01:34:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:34:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror_ns2-test.status
2024-05-08T01:34:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns2/test
2024-05-08T01:34:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror_ns2-test.status
2024-05-08T01:34:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:34:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:34:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:34:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:34:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:34:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror_ns2-test.status
2024-05-08T01:34:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:34:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:34:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror/ns1 test
2024-05-08T01:34:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:34:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns1 test up+replaying primary_position
2024-05-08T01:34:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:34:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:34:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:34:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:34:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror/ns1 test up+replaying primary_position ''
2024-05-08T01:34:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:34:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:34:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:34:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:34:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:34:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/ns1/test
2024-05-08T01:34:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:34:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   bf89f9df-5819-494d-8888-8c38d9847f78
2024-05-08T01:34:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:34:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}
2024-05-08T01:34:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:34:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:34:34
2024-05-08T01:34:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:34:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:34:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: down+unknown
2024-05-08T01:34:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: status not found
2024-05-08T01:34:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: '
2024-05-08T01:34:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:34:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:34:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:34:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}
2024-05-08T01:34:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:34:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:34:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:34:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:34:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:34:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:34:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:34:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:34:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:34:34'
2024-05-08T01:34:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns1 test up+replaying '2024-05-08 01:34:34'
2024-05-08T01:34:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:34:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:34:34'
2024-05-08T01:34:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:34:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:34:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror/ns1 --verbose --format xml
2024-05-08T01:34:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>5865</service_id><instance_id>5872</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>5869</service_id><instance_id>5881</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>5876</service_id><instance_id>5889</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>5883</service_id><instance_id>5893</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>bf89f9df-5819-494d-8888-8c38d9847f78</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:1}}</description><daemon_service><service_id>5865</service_id><instance_id>5872</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:34:34</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>down+unknown</state><description>status not found</description><last_update></last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:34:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:34:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:34:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:34:34'
2024-05-08T01:34:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:34:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:34:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:34:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:34:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:34:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror/ns2 test
2024-05-08T01:34:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:34:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns2 test up+replaying primary_position
2024-05-08T01:34:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:34:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:34:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:34:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:34:42.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror/ns2 test up+replaying primary_position ''
2024-05-08T01:34:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:34:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:34:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:34:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:34:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:34:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/ns2/test
2024-05-08T01:34:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:34:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   9323a208-7ab1-450b-bc8b-d7543fc4ad3a
2024-05-08T01:34:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:34:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}
2024-05-08T01:34:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:34:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:34:34
2024-05-08T01:34:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:34:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:34:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: down+unknown
2024-05-08T01:34:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: status not found
2024-05-08T01:34:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: '
2024-05-08T01:34:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:34:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:34:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:34:42.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}
2024-05-08T01:34:42.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:34:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:34:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:34:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:34:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:34:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:34:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:34:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:34:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:34:34'
2024-05-08T01:34:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns2 test up+replaying '2024-05-08 01:34:34'
2024-05-08T01:34:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:34:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:34:34'
2024-05-08T01:34:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:34:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:34:42.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror/ns2 --verbose --format xml
2024-05-08T01:34:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>5865</service_id><instance_id>5872</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>5869</service_id><instance_id>5881</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>5876</service_id><instance_id>5889</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>5883</service_id><instance_id>5893</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>9323a208-7ab1-450b-bc8b-d7543fc4ad3a</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:1}}</description><daemon_service><service_id>5865</service_id><instance_id>5872</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:34:34</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>down+unknown</state><description>status not found</description><last_update></last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:34:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:34:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:34:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:34:34'
2024-05-08T01:34:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:34:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:34:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:34:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:34:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:34:42.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:42.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:42.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror/ns1 test
2024-05-08T01:34:42.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:42.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:42.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:34:42.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror/ns1-test.export
2024-05-08T01:34:42.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror/ns1-test.export
2024-05-08T01:34:42.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:34:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.export
2024-05-08T01:34:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror/ns1-test.export
2024-05-08T01:34:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror/ns1-test.export
2024-05-08T01:34:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:34:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror_ns1-test.export
2024-05-08T01:34:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror_ns1-test.export
2024-05-08T01:34:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/ns1/test /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.export
2024-05-08T01:34:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:34:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/ns1/test /tmp/tmp.3xddryFtVF/cluster1-mirror_ns1-test.export
2024-05-08T01:34:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:34:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror_ns1-test.export
2024-05-08T01:34:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror_ns1-test.export
2024-05-08T01:34:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror/ns2 test
2024-05-08T01:34:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:34:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror/ns2-test.export
2024-05-08T01:34:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror/ns2-test.export
2024-05-08T01:34:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:34:43.281 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror_ns2-test.export
2024-05-08T01:34:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror/ns2-test.export
2024-05-08T01:34:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:34:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror/ns2-test.export
2024-05-08T01:34:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror_ns2-test.export
2024-05-08T01:34:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror_ns2-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror_ns2-test.export
2024-05-08T01:34:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/ns2/test /tmp/tmp.3xddryFtVF/cluster2-mirror_ns2-test.export
2024-05-08T01:34:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:34:43.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/ns2/test /tmp/tmp.3xddryFtVF/cluster1-mirror_ns2-test.export
2024-05-08T01:34:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:34:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror_ns2-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror_ns2-test.export
2024-05-08T01:34:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror_ns2-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror_ns2-test.export
2024-05-08T01:34:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog ' - disable mirroring / delete image'
2024-05-08T01:34:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:34:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:34:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:34:43 - disable mirroring / delete image
2024-05-08T01:34:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:34:43 - disable mirroring / delete image
2024-05-08T01:34:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster2 mirror/ns1 test
2024-05-08T01:34:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:34:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:34:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster2 mirror/ns1 test
2024-05-08T01:34:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 snap purge mirror/ns1/test
2024-05-08T01:34:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rm mirror/ns1/test
2024-05-08T01:34:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:34:43.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:43.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ disable_mirror cluster2 mirror/ns2 test
2024-05-08T01:34:43.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:43.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/ns2/test
2024-05-08T01:34:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Mirroring disabled
2024-05-08T01:34:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror/ns1 test deleted
2024-05-08T01:34:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=deleted
2024-05-08T01:34:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:34:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:34:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror/ns1 test
2024-05-08T01:34:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/ns1/test
2024-05-08T01:34:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test: (2) No such file or directory
2024-05-08T01:34:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=
2024-05-08T01:34:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:34:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:34:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror/ns1 test deleted ''
2024-05-08T01:34:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:34:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:34:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:34:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:34:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror/ns1 test
2024-05-08T01:34:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/ns1/test
2024-05-08T01:34:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test: (2) No such file or directory
2024-05-08T01:34:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:34:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = deleted
2024-05-08T01:34:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror/ns2 test deleted
2024-05-08T01:34:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=deleted
2024-05-08T01:34:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:34:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:34:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror/ns2 test
2024-05-08T01:34:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns2
2024-05-08T01:34:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/ns2/test
2024-05-08T01:34:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=16f0ce472396
2024-05-08T01:34:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:34:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:34:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror/ns2 test deleted 16f0ce472396
2024-05-08T01:34:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:34:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=16f0ce472396
2024-05-08T01:34:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:34:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:34:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror/ns2 test
2024-05-08T01:34:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns2
2024-05-08T01:34:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:44.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/ns2/test
2024-05-08T01:34:44.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=16f0ce472396
2024-05-08T01:34:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 16f0ce472396
2024-05-08T01:34:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z 16f0ce472396 -o 16f0ce472396 = 16f0ce472396
2024-05-08T01:34:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:34:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = present
2024-05-08T01:34:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:34:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:34:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror/ns2 test deleted 16f0ce472396
2024-05-08T01:34:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:34:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=16f0ce472396
2024-05-08T01:34:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:34:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:34:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror/ns2 test
2024-05-08T01:34:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns2
2024-05-08T01:34:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/ns2/test
2024-05-08T01:34:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test: (2) No such file or directory
2024-05-08T01:34:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:34:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = deleted
2024-05-08T01:34:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster2 mirror/ns2 test
2024-05-08T01:34:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:34:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:34:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster2 mirror/ns2 test
2024-05-08T01:34:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns2
2024-05-08T01:34:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 snap purge mirror/ns2/test
2024-05-08T01:34:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rm mirror/ns2/test
2024-05-08T01:34:45.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:34:45.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:45.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog ' - data pool'
2024-05-08T01:34:45.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:34:45.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:34:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:34:45 - data pool
2024-05-08T01:34:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:34:45 - data pool
2024-05-08T01:34:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ dp_image=test_data_pool
2024-05-08T01:34:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image_and_enable_mirror cluster2 mirror/ns1 test_data_pool journal 128 --data-pool mirror_parent
2024-05-08T01:34:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:34:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:34:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n journal ']'
2024-05-08T01:34:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image cluster2 mirror/ns1 test_data_pool 128 --data-pool mirror_parent
2024-05-08T01:34:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:34:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=128
2024-05-08T01:34:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 128 ']'
2024-05-08T01:34:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=128
2024-05-08T01:34:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --data-pool mirror_parent mirror/ns1/test_data_pool
2024-05-08T01:34:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ is_pool_mirror_mode_image mirror/ns1
2024-05-08T01:34:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' pool = image ']'
2024-05-08T01:34:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ case "${pool}" in
2024-05-08T01:34:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:34:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_data_pool cluster2 mirror/ns1 test_data_pool
2024-05-08T01:34:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:34:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 info mirror/ns1/test_data_pool
2024-05-08T01:34:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ awk '$1 == "data_pool:" {print $2}'
2024-05-08T01:34:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ data_pool=mirror_parent
2024-05-08T01:34:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test mirror_parent = mirror_parent
2024-05-08T01:34:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror/ns1 test_data_pool
2024-05-08T01:34:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:34:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror/ns1 test_data_pool started
2024-05-08T01:34:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:34:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:34:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:34:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:34:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror/ns1 test_data_pool started
2024-05-08T01:34:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:34:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:34:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:34:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:34:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/ns1/test_data_pool
2024-05-08T01:34:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:34:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:34:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:34:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:34:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:34:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:34:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:34:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test_data_pool
2024-05-08T01:34:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:34:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:34:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:34:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:34:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:34:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:34:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:34:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:34:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:34:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:34:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test_data_pool
2024-05-08T01:34:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool']
2024-05-08T01:34:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test_data_pool
2024-05-08T01:34:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:34:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:34:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:34:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:34:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:34:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:34:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:34:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:34:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:34:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:34:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:34:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:34:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:34:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test_data_pool
2024-05-08T01:34:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool']
2024-05-08T01:34:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/ns1/test_data_pool
2024-05-08T01:34:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:34:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:34:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:34:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:34:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:34:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:34:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:34:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:34:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:34:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:34:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:34:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:34:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:34:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:34:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:34:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:34:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:34:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:34:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:34:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns1/test_data_pool
2024-05-08T01:34:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool']
2024-05-08T01:34:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/ns1/test_data_pool
2024-05-08T01:34:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:34:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:34:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:34:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:34:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:34:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:34:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:34:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:34:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:34:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:34:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:34:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:34:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:34:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:34:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:34:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:34:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:34:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:34:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:34:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns1/test_data_pool
2024-05-08T01:34:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool']
2024-05-08T01:34:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 1
2024-05-08T01:34:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:34:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:34:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test_data_pool
2024-05-08T01:34:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:34:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:34:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:34:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:34:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:34:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:34:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:34:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:34:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:34:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:34:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test_data_pool
2024-05-08T01:34:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool']
2024-05-08T01:34:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test_data_pool
2024-05-08T01:34:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:34:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:34:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:34:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:34:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:34:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:34:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:34:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:34:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:34:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:34:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:34:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:34:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:34:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:34:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:34:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:34:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:34:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:34:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:34:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test_data_pool
2024-05-08T01:34:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool']
2024-05-08T01:34:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/ns1/test_data_pool
2024-05-08T01:34:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:34:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:34:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:34:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:34:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:34:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:34:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:34:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:34:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:34:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:34:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:34:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:34:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:34:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:34:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:34:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:34:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:34:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:34:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:34:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns1/test_data_pool
2024-05-08T01:34:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool']
2024-05-08T01:34:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/ns1/test_data_pool
2024-05-08T01:34:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:34:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:34:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:34:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:34:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:34:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:34:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:34:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:34:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:34:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:34:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:34:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:34:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:34:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:34:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:34:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:34:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:34:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:34:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:34:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns1/test_data_pool
2024-05-08T01:34:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool']
2024-05-08T01:34:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:34:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:34:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 2
2024-05-08T01:34:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:34:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:34:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test_data_pool
2024-05-08T01:34:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:34:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:34:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:34:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:34:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:34:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:34:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:34:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:34:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:34:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:34:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:34:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:34:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:34:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:34:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:34:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:34:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:34:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:34:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:34:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:34:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test_data_pool
2024-05-08T01:34:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:34:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:34:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:34:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:34:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:34:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:34:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:34:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_data_pool cluster1 mirror/ns1 test_data_pool
2024-05-08T01:34:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:34:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ awk '$1 == "data_pool:" {print $2}'
2024-05-08T01:34:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 info mirror/ns1/test_data_pool
2024-05-08T01:34:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ data_pool=mirror_parent
2024-05-08T01:34:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test mirror_parent = mirror_parent
2024-05-08T01:34:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror/ns1 test_data_pool 100
2024-05-08T01:34:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:34:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=100
2024-05-08T01:34:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:34:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:34:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/ns1/test_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand
2024-05-08T01:34:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 409600 pattern random
2024-05-08T01:34:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:34:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 100   ops/sec: 1923.13   bytes/sec: 7.5 MiB/s
2024-05-08T01:34:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns1 test_data_pool
2024-05-08T01:34:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:34:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:34:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:34:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror/ns1 test_data_pool
2024-05-08T01:34:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:34:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:34:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:34:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:34:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:34:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:34:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:34:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror/ns1 test_data_pool
2024-05-08T01:34:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:34:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:34:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test_data_pool ']'
2024-05-08T01:34:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/ns1/test_data_pool'
2024-05-08T01:34:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool
2024-05-08T01:34:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:34:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:34:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool
2024-05-08T01:34:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:34:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:34:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:34:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:34:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:34:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:34:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:34:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:34:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:34:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:34:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:34:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:34:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:34:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:34:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:34:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:34:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:34:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool
2024-05-08T01:34:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror/ns1 test_data_pool
2024-05-08T01:34:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:34:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror/ns1 test_data_pool ''
2024-05-08T01:34:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:34:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:34:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror/ns1-test_data_pool.status
2024-05-08T01:34:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror/ns1-test_data_pool.status
2024-05-08T01:34:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:34:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test_data_pool.status
2024-05-08T01:34:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool
2024-05-08T01:34:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test_data_pool.status
2024-05-08T01:34:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:34:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:34:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:34:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:34:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=2, entry_tid=12], [object_number=3, tag_tid=2, entry_tid=11], [object_number=2, tag_tid=2, entry_tid=10], [object_number=1, tag_tid=2, entry_tid=9]]], state=connected]
2024-05-08T01:34:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test_data_pool.status
2024-05-08T01:34:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:34:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool
2024-05-08T01:34:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:34:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror/ns1 test_data_pool '..*'
2024-05-08T01:34:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:34:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:34:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror/ns1-test_data_pool.status
2024-05-08T01:34:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror/ns1-test_data_pool.status
2024-05-08T01:34:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:34:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test_data_pool.status
2024-05-08T01:34:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool
2024-05-08T01:34:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test_data_pool.status
2024-05-08T01:34:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:34:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:34:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:34:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:34:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=0, tag_tid=2, entry_tid=12], [object_number=3, tag_tid=2, entry_tid=11], [object_number=2, tag_tid=2, entry_tid=10], [object_number=1, tag_tid=2, entry_tid=9]]], state=connected]
2024-05-08T01:34:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test_data_pool.status
2024-05-08T01:34:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=0, tag_tid=2, entry_tid=12'
2024-05-08T01:34:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=0, tag_tid=2, entry_tid=12'
2024-05-08T01:34:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=2, entry_tid=12' '!=' ''
2024-05-08T01:34:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:34:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=0, tag_tid=2, entry_tid=12' = ''
2024-05-08T01:34:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=0, tag_tid=2, entry_tid=12'
2024-05-08T01:34:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:34:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:34:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:34:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=2
2024-05-08T01:34:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=2, entry_tid=12'
2024-05-08T01:34:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:34:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:34:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=2
2024-05-08T01:34:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:34:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:34:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:34:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=101
2024-05-08T01:34:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=0, tag_tid=2, entry_tid=12'
2024-05-08T01:34:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:34:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:34:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=12
2024-05-08T01:34:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 2 = 2 -a 101 -le 12
2024-05-08T01:34:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:34:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:34:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:34:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror/ns1 test_data_pool
2024-05-08T01:34:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:34:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:34:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n test_data_pool ']'
2024-05-08T01:34:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/ns1/test_data_pool'
2024-05-08T01:34:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool
2024-05-08T01:34:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:34:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:34:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:34:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:34:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:34:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:34:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:34:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool
2024-05-08T01:34:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:34:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:34:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:34:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:34:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:34:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:34:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:34:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:34:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:34:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:34:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:34:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:34:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:34:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:34:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:34:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:34:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:34:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:34:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:34:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:34:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:34:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:34:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:34:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:34:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:34:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:34:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:34:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool
2024-05-08T01:34:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror/ns1 test_data_pool
2024-05-08T01:34:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:34:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror/ns1 test_data_pool ''
2024-05-08T01:34:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:34:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:34:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror/ns1-test_data_pool.status
2024-05-08T01:34:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror/ns1-test_data_pool.status
2024-05-08T01:34:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:34:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test_data_pool.status
2024-05-08T01:34:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool
2024-05-08T01:34:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test_data_pool.status
2024-05-08T01:34:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:34:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:34:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:34:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:34:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:34:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test_data_pool.status
2024-05-08T01:34:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:34:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool
2024-05-08T01:34:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:34:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror/ns1 test_data_pool '..*'
2024-05-08T01:34:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:34:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror/ns1
2024-05-08T01:34:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test_data_pool
2024-05-08T01:34:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:34:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror/ns1-test_data_pool.status
2024-05-08T01:34:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror/ns1-test_data_pool.status
2024-05-08T01:34:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:34:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test_data_pool.status
2024-05-08T01:34:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool
2024-05-08T01:34:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test_data_pool.status
2024-05-08T01:34:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:34:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:34:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:34:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:34:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected]
2024-05-08T01:34:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test_data_pool.status
2024-05-08T01:34:52.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:34:52.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101'
2024-05-08T01:34:52.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:52.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror/ns1 test_data_pool
2024-05-08T01:34:52.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:52.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:34:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:34:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns1 test_data_pool up+replaying primary_position
2024-05-08T01:34:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:34:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:34:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:34:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:34:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:34:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:34:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror/ns1 test_data_pool up+replaying primary_position ''
2024-05-08T01:34:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:34:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:34:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:34:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:34:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:34:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:34:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/ns1/test_data_pool
2024-05-08T01:34:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test_data_pool:
2024-05-08T01:34:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   d515ce50-3c18-4c16-b546-590c2cac2634
2024-05-08T01:34:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:34:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}
2024-05-08T01:34:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:34:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:34:48
2024-05-08T01:34:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:34:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:34:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: down+unknown
2024-05-08T01:34:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: status not found
2024-05-08T01:34:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: '
2024-05-08T01:34:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:34:53.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:34:53.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:34:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}
2024-05-08T01:34:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:34:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:34:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:34:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:34:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:34:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:34:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:34:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:34:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:34:48'
2024-05-08T01:34:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns1 test_data_pool up+replaying '2024-05-08 01:34:48'
2024-05-08T01:34:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:34:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:34:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:34:48'
2024-05-08T01:34:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:34:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:34:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror/ns1 --verbose --format xml
2024-05-08T01:34:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>OK</health><daemon_health>OK</daemon_health><image_health>OK</image_health><states><replaying>1</replaying></states></summary><daemons><daemon><service_id>5865</service_id><instance_id>5872</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>5869</service_id><instance_id>5881</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>5876</service_id><instance_id>5889</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>5883</service_id><instance_id>5893</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test_data_pool</name><global_id>d515ce50-3c18-4c16-b546-590c2cac2634</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:1}}</description><daemon_service><service_id>5865</service_id><instance_id>5872</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:34:48</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>down+unknown</state><description>status not found</description><last_update></last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:34:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:34:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/last_update'
2024-05-08T01:34:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:34:48'
2024-05-08T01:34:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/state'
2024-05-08T01:34:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:34:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:34:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:34:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:34:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror/ns1 test_data_pool
2024-05-08T01:34:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:34:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:34:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror/ns1-test_data_pool.export
2024-05-08T01:34:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror/ns1-test_data_pool.export
2024-05-08T01:34:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:34:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test_data_pool.export
2024-05-08T01:34:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror/ns1-test_data_pool.export
2024-05-08T01:34:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:34:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror/ns1-test_data_pool.export
2024-05-08T01:34:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror_ns1-test_data_pool.export
2024-05-08T01:34:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test_data_pool.export /tmp/tmp.3xddryFtVF/cluster1-mirror_ns1-test_data_pool.export
2024-05-08T01:34:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/ns1/test_data_pool /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test_data_pool.export
2024-05-08T01:34:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:34:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/ns1/test_data_pool /tmp/tmp.3xddryFtVF/cluster1-mirror_ns1-test_data_pool.export
2024-05-08T01:34:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:34:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test_data_pool.export /tmp/tmp.3xddryFtVF/cluster1-mirror_ns1-test_data_pool.export
2024-05-08T01:34:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror_ns1-test_data_pool.export /tmp/tmp.3xddryFtVF/cluster1-mirror_ns1-test_data_pool.export
2024-05-08T01:34:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster2 mirror/ns1 test_data_pool
2024-05-08T01:34:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:34:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:34:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:34:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster2 mirror/ns1 test_data_pool
2024-05-08T01:34:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:34:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror/ns1
2024-05-08T01:34:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test_data_pool
2024-05-08T01:34:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 snap purge mirror/ns1/test_data_pool
2024-05-08T01:34:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rm mirror/ns1/test_data_pool
2024-05-08T01:34:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:34:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: simple image resync'
2024-05-08T01:34:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:34:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:34:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:34:53 TEST: simple image resync
2024-05-08T01:34:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:34:53 TEST: simple image resync
2024-05-08T01:34:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ request_resync_image cluster1 mirror test image_id
2024-05-08T01:34:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id_var_name=image_id
2024-05-08T01:34:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:34:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:34:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:34:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval 'image_id='\''16f0e3b9669d'\'''
2024-05-08T01:34:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ image_id=16f0e3b9669d
2024-05-08T01:34:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval 'test -n "$image_id"'
2024-05-08T01:34:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -n 16f0e3b9669d
2024-05-08T01:34:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/test
2024-05-08T01:34:53.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Flagged image for resync from primary
2024-05-08T01:34:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test deleted 16f0e3b9669d
2024-05-08T01:34:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=deleted
2024-05-08T01:34:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=16f0e3b9669d
2024-05-08T01:34:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:34:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 16f0e3b9669d
2024-05-08T01:34:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:34:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:34:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test deleted 16f0e3b9669d
2024-05-08T01:34:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:34:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=16f0e3b9669d
2024-05-08T01:34:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:34:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:34:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:34:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:34:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:34:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test: (2) No such file or directory
2024-05-08T01:34:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:34:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = deleted
2024-05-08T01:34:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:34:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test present
2024-05-08T01:34:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=present
2024-05-08T01:34:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:34:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:34:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:34:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:34:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:34:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test: (2) No such file or directory
2024-05-08T01:34:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=
2024-05-08T01:34:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:34:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:34:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test present ''
2024-05-08T01:34:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:34:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:34:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:34:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:34:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:34:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:34:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:34:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test: (2) No such file or directory
2024-05-08T01:34:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:34:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:34:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:34:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test present ''
2024-05-08T01:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:34:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:34:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:34:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:34:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test: (2) No such file or directory
2024-05-08T01:34:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:34:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:34:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:34:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:34:57.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test present ''
2024-05-08T01:34:57.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:34:57.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:34:57.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:34:57.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:34:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:34:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:34:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:34:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:34:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:34:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:34:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:34:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:34:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:34:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test: (2) No such file or directory
2024-05-08T01:34:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:34:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:34:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:34:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:34:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 4
2024-05-08T01:35:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test present ''
2024-05-08T01:35:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:35:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:35:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:01.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:35:01.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:01.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:01.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:35:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:35:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=16f0c46cd6c4
2024-05-08T01:35:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 16f0c46cd6c4
2024-05-08T01:35:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z '' -o '' = 16f0c46cd6c4
2024-05-08T01:35:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:35:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = present
2024-05-08T01:35:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:35:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:35:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:35:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:35:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:35:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:35:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:35:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:35:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:35:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:35:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:35:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:35:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:35:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:35:02.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:35:02.561 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:35:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:35:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:35:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:35:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:35:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:35:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:35:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:35:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:35:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:35:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:35:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:35:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:02.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:02.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:02.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:02.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:35:02.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:35:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:35:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:35:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:35:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:35:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:35:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:35:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test
2024-05-08T01:35:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:35:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position
2024-05-08T01:35:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:35:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:35:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:35:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position ''
2024-05-08T01:35:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:35:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:35:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:35:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:35:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:35:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:35:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:35:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   ced542d9-0ad3-4794-a43d-8d9a38351e02
2024-05-08T01:35:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:35:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:35:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:35:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:34:59
2024-05-08T01:35:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:35:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:35:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: down+unknown
2024-05-08T01:35:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: status not found
2024-05-08T01:35:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: '
2024-05-08T01:35:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:35:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:35:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:35:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:35:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:35:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:35:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:35:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:35:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:35:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:35:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:35:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:35:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:34:59'
2024-05-08T01:35:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:34:59'
2024-05-08T01:35:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:35:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:34:59'
2024-05-08T01:35:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:35:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:35:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:35:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>5865</service_id><instance_id>5872</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>5869</service_id><instance_id>5881</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>5876</service_id><instance_id>5889</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>5883</service_id><instance_id>5893</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>ced542d9-0ad3-4794-a43d-8d9a38351e02</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:1},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><daemon_service><service_id>5865</service_id><instance_id>5872</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:34:59</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>down+unknown</state><description>status not found</description><last_update></last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:35:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:35:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:35:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:34:59'
2024-05-08T01:35:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:35:03.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:35:03.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:35:03.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:35:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:35:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror test
2024-05-08T01:35:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:35:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror-test.export
2024-05-08T01:35:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror-test.export
2024-05-08T01:35:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:35:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.export
2024-05-08T01:35:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror-test.export
2024-05-08T01:35:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror-test.export
2024-05-08T01:35:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:35:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:35:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:35:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export
2024-05-08T01:35:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:35:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:35:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:35:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:35:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:35:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -z '' ']'
2024-05-08T01:35:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: image resync while replayer is stopped'
2024-05-08T01:35:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:35:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:35:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:35:04 TEST: image resync while replayer is stopped
2024-05-08T01:35:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:35:04 TEST: image resync while replayer is stopped
2024-05-08T01:35:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror stop mirror/test
2024-05-08T01:35:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:35:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:35:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:35:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:35:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:35:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:35:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:35:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror/test
2024-05-08T01:35:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:35:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:35:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:35:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:35:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:35:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:35:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:35:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:35:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:35:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:35:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:35:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:35:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:35:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:35:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:35:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror/test
2024-05-08T01:35:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror test
2024-05-08T01:35:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped
2024-05-08T01:35:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:35:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:35:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test stopped
2024-05-08T01:35:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:35:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:35:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:35:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:35:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:35:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:35:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:35:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:35:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:05.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:35:05.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:35:05.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:35:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:35:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:35:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:35:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:35:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:35:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:35:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:35:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:35:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:35:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:35:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:35:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:35:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:35:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:35:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:35:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:35:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:35:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ request_resync_image cluster1 mirror test image_id
2024-05-08T01:35:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id_var_name=image_id
2024-05-08T01:35:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:35:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:35:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:35:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval 'image_id='\''16f0c46cd6c4'\'''
2024-05-08T01:35:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ image_id=16f0c46cd6c4
2024-05-08T01:35:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval 'test -n "$image_id"'
2024-05-08T01:35:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -n 16f0c46cd6c4
2024-05-08T01:35:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/test
2024-05-08T01:35:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Flagged image for resync from primary
2024-05-08T01:35:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror start mirror/test
2024-05-08T01:35:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:35:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:35:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:35:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:35:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:35:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:35:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:35:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/test
2024-05-08T01:35:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:35:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:35:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:35:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:35:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:35:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:35:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:35:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:35:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:35:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:35:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:35:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:35:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:35:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:35:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:35:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/test
2024-05-08T01:35:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test deleted 16f0c46cd6c4
2024-05-08T01:35:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=deleted
2024-05-08T01:35:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=16f0c46cd6c4
2024-05-08T01:35:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:35:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 16f0c46cd6c4
2024-05-08T01:35:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:35:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test deleted 16f0c46cd6c4
2024-05-08T01:35:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:35:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=16f0c46cd6c4
2024-05-08T01:35:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:35:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:35:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:35:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:06.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image test: (2) No such file or directory
2024-05-08T01:35:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:35:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = deleted
2024-05-08T01:35:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror start mirror/test
2024-05-08T01:35:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:35:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:35:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:35:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:35:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:35:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:35:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:35:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/test
2024-05-08T01:35:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:35:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:35:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:35:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:35:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:35:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:35:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:35:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:35:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:35:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:35:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:35:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:35:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:35:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:35:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:35:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/test
2024-05-08T01:35:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test present
2024-05-08T01:35:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=present
2024-05-08T01:35:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:35:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:35:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:35:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:35:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:35:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:06.570 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=16f06ea4743f
2024-05-08T01:35:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:35:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test present 16f06ea4743f
2024-05-08T01:35:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:35:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=16f06ea4743f
2024-05-08T01:35:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:35:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:35:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:35:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=16f06ea4743f
2024-05-08T01:35:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 16f06ea4743f
2024-05-08T01:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z 16f06ea4743f -o 16f06ea4743f = 16f06ea4743f
2024-05-08T01:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = present
2024-05-08T01:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:35:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:35:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:35:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:35:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:35:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:35:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:35:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:35:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:35:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:35:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:35:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:35:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:35:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:35:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:35:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:35:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:35:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:35:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:35:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:35:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:35:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:35:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:35:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:35:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:35:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:07.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:07.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:07.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:35:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:35:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:35:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:35:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:35:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:35:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:35:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:35:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test
2024-05-08T01:35:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:35:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position
2024-05-08T01:35:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:35:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:35:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:35:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position ''
2024-05-08T01:35:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:35:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:35:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:35:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:35:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:35:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:35:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:35:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   ced542d9-0ad3-4794-a43d-8d9a38351e02
2024-05-08T01:35:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:35:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}}
2024-05-08T01:35:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:35:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:35:07
2024-05-08T01:35:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:35:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:35:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: down+unknown
2024-05-08T01:35:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: status not found
2024-05-08T01:35:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: '
2024-05-08T01:35:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:35:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:35:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:35:08.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}}
2024-05-08T01:35:08.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:35:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:35:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:35:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:35:08.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:35:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:35:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:35:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:35:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:35:07'
2024-05-08T01:35:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:35:07'
2024-05-08T01:35:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:35:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:35:07'
2024-05-08T01:35:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:35:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:35:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:35:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>5865</service_id><instance_id>5872</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>5869</service_id><instance_id>5881</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>5876</service_id><instance_id>5889</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>5883</service_id><instance_id>5893</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>ced542d9-0ad3-4794-a43d-8d9a38351e02</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:3}}</description><daemon_service><service_id>5865</service_id><instance_id>5872</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:35:07</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>down+unknown</state><description>status not found</description><last_update></last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:35:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:35:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:35:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:35:07'
2024-05-08T01:35:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:35:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:35:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:35:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:35:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:35:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror test
2024-05-08T01:35:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:35:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror-test.export
2024-05-08T01:35:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror-test.export
2024-05-08T01:35:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:35:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.export
2024-05-08T01:35:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror-test.export
2024-05-08T01:35:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror-test.export
2024-05-08T01:35:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:35:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:35:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:35:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export
2024-05-08T01:35:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:35:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:35:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:35:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:35:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:35:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: request image resync while daemon is offline'
2024-05-08T01:35:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:35:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:35:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:35:09 TEST: request image resync while daemon is offline
2024-05-08T01:35:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:35:09 TEST: request image resync while daemon is offline
2024-05-08T01:35:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirrors cluster1
2024-05-08T01:35:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:35:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:35:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster1:0 ''
2024-05-08T01:35:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:35:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:35:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:35:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster1:0
2024-05-08T01:35:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1:0
2024-05-08T01:35:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:35:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:35:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster1
2024-05-08T01:35:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster1
2024-05-08T01:35:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:35:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:35:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file'
2024-05-08T01:35:09.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid
2024-05-08T01:35:09.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid
2024-05-08T01:35:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=66684
2024-05-08T01:35:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 66684 ']'
2024-05-08T01:35:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 66684
2024-05-08T01:35:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:35:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:10.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:35:10.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=66684 '$2 == pid {print; exit 1}'
2024-05-08T01:35:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:35:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:35:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=66684 '$2 == pid {print; exit 1}'
2024-05-08T01:35:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:35:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:35:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:35:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:35:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:35:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster2
2024-05-08T01:35:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:35:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:35:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:35:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:35:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:10.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:10.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster1:0
2024-05-08T01:35:10.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:35:10.859 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file'
2024-05-08T01:35:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid
2024-05-08T01:35:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid
2024-05-08T01:35:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster1:1 ''
2024-05-08T01:35:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:35:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:35:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:35:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster1:1
2024-05-08T01:35:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1:1
2024-05-08T01:35:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:35:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:35:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:35:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:35:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:35:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster1
2024-05-08T01:35:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster1
2024-05-08T01:35:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:35:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:35:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file'
2024-05-08T01:35:10.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid
2024-05-08T01:35:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid
2024-05-08T01:35:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=66712
2024-05-08T01:35:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 66712 ']'
2024-05-08T01:35:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 66712
2024-05-08T01:35:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:35:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:35:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=66712 '$2 == pid {print; exit 1}'
2024-05-08T01:35:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:35:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:35:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=66712 '$2 == pid {print; exit 1}'
2024-05-08T01:35:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:35:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:35:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:35:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:35:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:35:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:35:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:35:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:35:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:35:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:35:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:35:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:35:11.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:35:11.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:35:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster2
2024-05-08T01:35:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:35:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:35:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:35:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:35:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:35:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:35:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:35:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:35:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:35:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:35:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:35:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:35:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:35:11.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster1:1
2024-05-08T01:35:11.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:35:11.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:35:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:35:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:35:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:35:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:35:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:35:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file'
2024-05-08T01:35:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid
2024-05-08T01:35:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid
2024-05-08T01:35:12.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:12.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster1:2 ''
2024-05-08T01:35:12.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:35:12.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:35:12.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:12.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:35:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster1:2
2024-05-08T01:35:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1:2
2024-05-08T01:35:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:35:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:35:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:35:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:35:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:35:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster1
2024-05-08T01:35:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster1
2024-05-08T01:35:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:35:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:35:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file'
2024-05-08T01:35:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid
2024-05-08T01:35:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid
2024-05-08T01:35:12.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=66756
2024-05-08T01:35:12.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 66756 ']'
2024-05-08T01:35:12.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 66756
2024-05-08T01:35:12.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:35:12.042 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:35:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=66756 '$2 == pid {print; exit 1}'
2024-05-08T01:35:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:35:13.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:35:13.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=66756 '$2 == pid {print; exit 1}'
2024-05-08T01:35:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:35:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:35:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:35:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:35:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:35:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:35:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:35:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:35:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:35:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:35:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:35:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:35:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:35:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:35:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster2
2024-05-08T01:35:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:35:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:35:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:35:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:35:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:35:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:35:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:35:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:35:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:35:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:35:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:35:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:35:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:35:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster1:2
2024-05-08T01:35:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:35:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:35:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:35:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:35:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:35:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:35:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:35:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file'
2024-05-08T01:35:13.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid
2024-05-08T01:35:13.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid
2024-05-08T01:35:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster1:3 ''
2024-05-08T01:35:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:35:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:35:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:35:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster1:3
2024-05-08T01:35:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1:3
2024-05-08T01:35:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:35:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:35:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:35:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:35:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:35:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster1
2024-05-08T01:35:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster1
2024-05-08T01:35:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:35:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:35:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file'
2024-05-08T01:35:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid
2024-05-08T01:35:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid
2024-05-08T01:35:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=66816
2024-05-08T01:35:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 66816 ']'
2024-05-08T01:35:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 66816
2024-05-08T01:35:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:35:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:35:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=66816 '$2 == pid {print; exit 1}'
2024-05-08T01:35:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:35:14.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:35:14.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=66816 '$2 == pid {print; exit 1}'
2024-05-08T01:35:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:35:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:35:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:35:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:35:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:35:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:35:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:35:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:35:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:35:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:35:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:35:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:35:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:35:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:35:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster2
2024-05-08T01:35:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:35:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:35:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:35:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:35:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:35:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:35:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:35:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:35:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:35:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:35:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:35:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:35:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:35:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster1:3
2024-05-08T01:35:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:35:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:35:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:35:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:35:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:35:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:35:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:35:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file'
2024-05-08T01:35:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid
2024-05-08T01:35:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid
2024-05-08T01:35:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ request_resync_image cluster1 mirror test image_id
2024-05-08T01:35:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id_var_name=image_id
2024-05-08T01:35:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:35:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:35:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:35:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval 'image_id='\''16f06ea4743f'\'''
2024-05-08T01:35:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ image_id=16f06ea4743f
2024-05-08T01:35:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval 'test -n "$image_id"'
2024-05-08T01:35:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -n 16f06ea4743f
2024-05-08T01:35:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/test
2024-05-08T01:35:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Flagged image for resync from primary
2024-05-08T01:35:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirrors cluster1
2024-05-08T01:35:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:35:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster1:0
2024-05-08T01:35:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:35:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:35:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:35:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:35:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:35:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:35:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:35:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:35:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:35:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:35:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster1:1
2024-05-08T01:35:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:35:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:35:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:35:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:1
2024-05-08T01:35:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:35:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:35:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:35:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:35:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster1:1 ']'
2024-05-08T01:35:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:35:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:35:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:35:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:35:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster1:2
2024-05-08T01:35:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:2
2024-05-08T01:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster1:2 ']'
2024-05-08T01:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:35:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster1:3
2024-05-08T01:35:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:35:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:35:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:35:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:3
2024-05-08T01:35:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:35:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:35:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:35:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:35:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster1:3 ']'
2024-05-08T01:35:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:35:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:35:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:35:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:35:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test deleted 16f06ea4743f
2024-05-08T01:35:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=deleted
2024-05-08T01:35:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=16f06ea4743f
2024-05-08T01:35:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:35:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 16f06ea4743f
2024-05-08T01:35:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:35:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test deleted 16f06ea4743f
2024-05-08T01:35:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:35:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=16f06ea4743f
2024-05-08T01:35:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:35:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:35:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:35:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:14.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=16f06ea4743f
2024-05-08T01:35:14.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 16f06ea4743f
2024-05-08T01:35:14.737 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z 16f06ea4743f -o 16f06ea4743f = 16f06ea4743f
2024-05-08T01:35:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:35:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = present
2024-05-08T01:35:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test deleted 16f06ea4743f
2024-05-08T01:35:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:35:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=16f06ea4743f
2024-05-08T01:35:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:35:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:35:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:35:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=16f06ea4743f
2024-05-08T01:35:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 16f06ea4743f
2024-05-08T01:35:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z 16f06ea4743f -o 16f06ea4743f = 16f06ea4743f
2024-05-08T01:35:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:35:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = present
2024-05-08T01:35:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:35:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test deleted 16f06ea4743f
2024-05-08T01:35:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:35:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=16f06ea4743f
2024-05-08T01:35:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:35:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:35:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:35:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:17.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=16f06ea4743f
2024-05-08T01:35:17.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 16f06ea4743f
2024-05-08T01:35:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z 16f06ea4743f -o 16f06ea4743f = 16f06ea4743f
2024-05-08T01:35:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:35:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = present
2024-05-08T01:35:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 4
2024-05-08T01:35:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test deleted 16f06ea4743f
2024-05-08T01:35:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:35:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=16f06ea4743f
2024-05-08T01:35:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:35:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:35:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:35:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=179d8fa19939
2024-05-08T01:35:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 179d8fa19939
2024-05-08T01:35:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z 16f06ea4743f -o 16f06ea4743f = 179d8fa19939
2024-05-08T01:35:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = deleted
2024-05-08T01:35:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror test present
2024-05-08T01:35:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=present
2024-05-08T01:35:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:35:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:35:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:35:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:35:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:35:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=179d8fa19939
2024-05-08T01:35:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:35:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror test present 179d8fa19939
2024-05-08T01:35:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:35:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=179d8fa19939
2024-05-08T01:35:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror test
2024-05-08T01:35:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=test
2024-05-08T01:35:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/test
2024-05-08T01:35:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=179d8fa19939
2024-05-08T01:35:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 179d8fa19939
2024-05-08T01:35:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z 179d8fa19939 -o 179d8fa19939 = 179d8fa19939
2024-05-08T01:35:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:35:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = present
2024-05-08T01:35:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror test
2024-05-08T01:35:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror test started
2024-05-08T01:35:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:35:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:35:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror test started
2024-05-08T01:35:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:35:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:35:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:35:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test
2024-05-08T01:35:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:35:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:35:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:35:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:35:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:35:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:35:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:35:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test
2024-05-08T01:35:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:35:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:35:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:35:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:35:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:35:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:35:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:35:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:35:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:35:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:35:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test
2024-05-08T01:35:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:35:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:35:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:35:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:35:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:35:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:35:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:35:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror test
2024-05-08T01:35:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:35:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position
2024-05-08T01:35:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:35:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:35:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:35:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position ''
2024-05-08T01:35:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:35:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:35:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:35:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:35:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:35:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/test
2024-05-08T01:35:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='test:
2024-05-08T01:35:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   ced542d9-0ad3-4794-a43d-8d9a38351e02
2024-05-08T01:35:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:35:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}}
2024-05-08T01:35:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:35:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:35:20
2024-05-08T01:35:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:35:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:35:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: down+unknown
2024-05-08T01:35:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: status not found
2024-05-08T01:35:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: '
2024-05-08T01:35:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:35:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:35:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:35:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}}
2024-05-08T01:35:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:35:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:35:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:35:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:35:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:35:24.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:35:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:35:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:35:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:35:20'
2024-05-08T01:35:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 01:35:20'
2024-05-08T01:35:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:35:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:35:20'
2024-05-08T01:35:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:35:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:35:24.436 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:35:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>6038</service_id><instance_id>6045</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>6042</service_id><instance_id>6055</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>6049</service_id><instance_id>6059</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>6056</service_id><instance_id>6062</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>test</name><global_id>ced542d9-0ad3-4794-a43d-8d9a38351e02</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:3},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:4}}</description><daemon_service><service_id>6038</service_id><instance_id>6045</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:35:20</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>down+unknown</state><description>status not found</description><last_update></last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:35:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:35:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update'
2024-05-08T01:35:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:35:20'
2024-05-08T01:35:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state'
2024-05-08T01:35:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:35:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:35:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:35:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:35:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror test
2024-05-08T01:35:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:35:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror-test.export
2024-05-08T01:35:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror-test.export
2024-05-08T01:35:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:35:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror-test.export
2024-05-08T01:35:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror-test.export
2024-05-08T01:35:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror-test.export
2024-05-08T01:35:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:35:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:35:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:35:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export
2024-05-08T01:35:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:35:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:35:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:35:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:35:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-test.export /tmp/tmp.3xddryFtVF/cluster1-mirror-test.export
2024-05-08T01:35:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster2 mirror test
2024-05-08T01:35:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:35:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:35:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:35:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster2 mirror test
2024-05-08T01:35:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:35:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=test
2024-05-08T01:35:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 snap purge mirror/test
2024-05-08T01:35:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rm mirror/test
2024-05-08T01:35:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:35:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:35:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: client disconnect'
2024-05-08T01:35:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:35:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:35:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:35:25 TEST: client disconnect
2024-05-08T01:35:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:35:25 TEST: client disconnect
2024-05-08T01:35:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image=laggy
2024-05-08T01:35:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image cluster2 mirror laggy 128 --journal-object-size 64K
2024-05-08T01:35:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:35:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:35:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:35:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:35:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=128
2024-05-08T01:35:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 128 ']'
2024-05-08T01:35:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=128
2024-05-08T01:35:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:35:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --journal-object-size 64K mirror/laggy
2024-05-08T01:35:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror laggy 10
2024-05-08T01:35:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:35:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=10
2024-05-08T01:35:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:35:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:35:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/laggy --io-type write --io-size 4096 --io-threads 1 --io-total 40960 --io-pattern rand
2024-05-08T01:35:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 40960 pattern random
2024-05-08T01:35:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:35:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 10   ops/sec: 833.355   bytes/sec: 3.3 MiB/s
2024-05-08T01:35:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog ' - replay stopped after disconnect'
2024-05-08T01:35:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:35:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:35:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:35:25 - replay stopped after disconnect
2024-05-08T01:35:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:35:25 - replay stopped after disconnect
2024-05-08T01:35:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror laggy
2024-05-08T01:35:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror laggy started
2024-05-08T01:35:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:35:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:35:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror laggy started
2024-05-08T01:35:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:35:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:35:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:35:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy
2024-05-08T01:35:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:35:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:35:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:35:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:35:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:35:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:35:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:35:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy
2024-05-08T01:35:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:35:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:35:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:35:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:35:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:35:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:35:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:35:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:35:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:35:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:35:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy
2024-05-08T01:35:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/laggy']
2024-05-08T01:35:26.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:35:26.384 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:35:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy
2024-05-08T01:35:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:35:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:35:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:35:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:35:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:35:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:35:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:35:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:35:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:35:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:35:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:35:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:35:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:35:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:35:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:35:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:35:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:35:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:35:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:35:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:35:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:35:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:35:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:35:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy
2024-05-08T01:35:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/laggy']
2024-05-08T01:35:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:35:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:35:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/laggy
2024-05-08T01:35:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:35:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:35:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:35:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:35:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:35:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:35:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:35:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:35:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:35:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:35:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:35:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:35:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:35:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:35:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:35:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:35:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:35:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:35:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:35:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:35:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:35:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:35:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:35:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/laggy
2024-05-08T01:35:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/laggy']
2024-05-08T01:35:26.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:35:26.649 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:35:26.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:26.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/laggy
2024-05-08T01:35:26.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:35:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:35:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:35:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:35:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:35:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:35:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:35:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:35:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:35:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:35:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:35:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:35:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:35:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:35:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:35:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:35:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:35:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:35:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:35:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:35:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:35:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:35:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:35:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/laggy
2024-05-08T01:35:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/laggy']
2024-05-08T01:35:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:35:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:35:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 1
2024-05-08T01:35:27.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:35:27.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:35:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy
2024-05-08T01:35:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:35:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:35:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:35:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:35:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:35:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:35:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:35:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:35:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:35:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:35:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy
2024-05-08T01:35:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/laggy']
2024-05-08T01:35:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:35:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:35:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy
2024-05-08T01:35:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:35:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:35:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:35:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:35:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:35:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:35:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:35:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:35:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:1
2024-05-08T01:35:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:35:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:35:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:35:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:35:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:35:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:35:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:35:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:35:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:35:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:35:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:35:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:35:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:35:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:35:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy
2024-05-08T01:35:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/laggy']
2024-05-08T01:35:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:35:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:35:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/laggy
2024-05-08T01:35:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:35:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:35:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:35:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:35:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:35:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:35:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:35:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:2
2024-05-08T01:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:35:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:35:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:35:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:35:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:35:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/laggy
2024-05-08T01:35:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/laggy']
2024-05-08T01:35:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:35:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:35:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/laggy
2024-05-08T01:35:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:35:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:35:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:35:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:35:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:35:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:35:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:35:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:35:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:3
2024-05-08T01:35:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:35:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:35:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:35:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:35:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:35:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:35:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:35:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:35:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:35:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:35:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:35:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:35:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:35:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:35:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/laggy
2024-05-08T01:35:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:Invalid command: unused arguments: ['mirror/laggy']
2024-05-08T01:35:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd mirror status :  get status for rbd mirror
2024-05-08T01:35:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:admin_socket: invalid command
2024-05-08T01:35:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 2
2024-05-08T01:35:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:35:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:35:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy
2024-05-08T01:35:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:35:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:35:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:35:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:35:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:35:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:35:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:35:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:35:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:35:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:35:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy
2024-05-08T01:35:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:35:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:35:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:35:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:35:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:35:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:35:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:35:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror laggy
2024-05-08T01:35:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:35:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:35:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:35:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror laggy
2024-05-08T01:35:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:35:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:35:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:35:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:35:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:35:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:35:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:35:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror laggy
2024-05-08T01:35:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:35:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n laggy ']'
2024-05-08T01:35:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/laggy'
2024-05-08T01:35:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy
2024-05-08T01:35:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:35:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:35:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:35:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:35:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:35:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:35:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:35:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy
2024-05-08T01:35:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:35:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:35:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:35:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:35:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:35:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:35:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:35:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:35:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:35:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:35:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:35:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:35:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:35:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:35:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:35:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy
2024-05-08T01:35:30.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror laggy
2024-05-08T01:35:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy ''
2024-05-08T01:35:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:35:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:35:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:35:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:35:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:35:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:35:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:35:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:35:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:35:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:35:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:35:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror laggy
2024-05-08T01:35:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy '..*'
2024-05-08T01:35:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:35:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:35:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:35:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:35:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:35:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:35:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:35:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:35:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:35:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:35:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:30.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:35:30.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:35:30.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:30.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror laggy
2024-05-08T01:35:30.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:30.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:30.939 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:30.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy '..*'
2024-05-08T01:35:30.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:30.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:30.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:30.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:35:30.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:35:30.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:35:30.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:35:30.940 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:35:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:35:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:35:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:35:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:35:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:35:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3'
2024-05-08T01:35:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ disconnect_image cluster2 mirror laggy
2024-05-08T01:35:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:35:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 journal client disconnect --image mirror/laggy
2024-05-08T01:35:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:client d8559015-eaca-4ba8-a079-91f9b3432357 disconnected
2024-05-08T01:35:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror laggy
2024-05-08T01:35:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy '..*'
2024-05-08T01:35:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:35:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:35:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:35:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:35:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:35:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:35:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:35:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:35:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected]
2024-05-08T01:35:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=disconnected]
2024-05-08T01:35:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z ''
2024-05-08T01:35:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror laggy
2024-05-08T01:35:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror laggy stopped
2024-05-08T01:35:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:35:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:35:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror laggy stopped
2024-05-08T01:35:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:35:32.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:35:32.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:35:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy
2024-05-08T01:35:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:35:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:35:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:35:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:35:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:35:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:35:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:35:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy
2024-05-08T01:35:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:35:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:35:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:35:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:35:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:35:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:35:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:35:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:35:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:35:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:35:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy
2024-05-08T01:35:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:35:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:35:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:35:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:35:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:35:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror laggy up+error disconnected
2024-05-08T01:35:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+error
2024-05-08T01:35:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=disconnected
2024-05-08T01:35:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:35:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror laggy up+error disconnected ''
2024-05-08T01:35:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+error
2024-05-08T01:35:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=disconnected
2024-05-08T01:35:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:35:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:35:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:35:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/laggy
2024-05-08T01:35:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='laggy:
2024-05-08T01:35:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   5ddea2ca-740c-4e60-ac31-ee957a3be11f
2024-05-08T01:35:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+error
2024-05-08T01:35:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: disconnected
2024-05-08T01:35:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:35:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:35:31
2024-05-08T01:35:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:35:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:35:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: down+unknown
2024-05-08T01:35:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: status not found
2024-05-08T01:35:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: '
2024-05-08T01:35:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+error'
2024-05-08T01:35:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+error
2024-05-08T01:35:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*disconnected'
2024-05-08T01:35:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: disconnected
2024-05-08T01:35:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:35:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+error
2024-05-08T01:35:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:35:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+error
2024-05-08T01:35:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:35:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:35:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:35:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:35:33.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:35:31'
2024-05-08T01:35:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror laggy up+error '2024-05-08 01:35:31'
2024-05-08T01:35:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+error
2024-05-08T01:35:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:35:31'
2024-05-08T01:35:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:35:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:35:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:35:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>6038</service_id><instance_id>6045</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>6042</service_id><instance_id>6055</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>6049</service_id><instance_id>6059</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>6056</service_id><instance_id>6062</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>laggy</name><global_id>5ddea2ca-740c-4e60-ac31-ee957a3be11f</global_id><state>up+error</state><description>disconnected</description><daemon_service><service_id>6038</service_id><instance_id>6045</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:35:31</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>down+unknown</state><description>status not found</description><last_update></last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:35:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:35:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/last_update'
2024-05-08T01:35:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:35:31'
2024-05-08T01:35:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/state'
2024-05-08T01:35:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+error
2024-05-08T01:35:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+error
2024-05-08T01:35:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+error
2024-05-08T01:35:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+error
2024-05-08T01:35:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog ' - replay started after resync requested'
2024-05-08T01:35:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:35:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:35:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:35:33 - replay started after resync requested
2024-05-08T01:35:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:35:33 - replay started after resync requested
2024-05-08T01:35:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ request_resync_image cluster1 mirror laggy image_id
2024-05-08T01:35:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id_var_name=image_id
2024-05-08T01:35:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:35:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:35:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval 'image_id='\''179d123dd051'\'''
2024-05-08T01:35:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ image_id=179d123dd051
2024-05-08T01:35:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval 'test -n "$image_id"'
2024-05-08T01:35:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -n 179d123dd051
2024-05-08T01:35:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/laggy
2024-05-08T01:35:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Flagged image for resync from primary
2024-05-08T01:35:33.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror laggy deleted 179d123dd051
2024-05-08T01:35:33.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:33.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:33.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=deleted
2024-05-08T01:35:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=179d123dd051
2024-05-08T01:35:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:35:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 179d123dd051
2024-05-08T01:35:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:35:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror laggy deleted 179d123dd051
2024-05-08T01:35:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:35:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=179d123dd051
2024-05-08T01:35:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:35:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:33.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:33.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:33.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:35:33.908 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=179d123dd051
2024-05-08T01:35:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 179d123dd051
2024-05-08T01:35:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z 179d123dd051 -o 179d123dd051 = 179d123dd051
2024-05-08T01:35:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:35:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = present
2024-05-08T01:35:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror laggy deleted 179d123dd051
2024-05-08T01:35:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:35:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=179d123dd051
2024-05-08T01:35:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:35:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:35:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image laggy: (2) No such file or directory
2024-05-08T01:35:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:35:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = deleted
2024-05-08T01:35:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror laggy present
2024-05-08T01:35:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=present
2024-05-08T01:35:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:35:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:35:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:35:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:35:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image laggy: (2) No such file or directory
2024-05-08T01:35:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=
2024-05-08T01:35:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:35:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror laggy present ''
2024-05-08T01:35:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:35:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:35:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:35:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:35:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image laggy: (2) No such file or directory
2024-05-08T01:35:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:35:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:35:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror laggy present ''
2024-05-08T01:35:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:35:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:35:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:35:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:35:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image laggy: (2) No such file or directory
2024-05-08T01:35:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:35:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:35:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:35:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror laggy present ''
2024-05-08T01:35:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:35:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:35:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:35:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:35:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image laggy: (2) No such file or directory
2024-05-08T01:35:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:35:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:35:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 4
2024-05-08T01:35:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror laggy present ''
2024-05-08T01:35:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:35:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:35:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:35:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:35:42.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=179d9a681d20
2024-05-08T01:35:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 179d9a681d20
2024-05-08T01:35:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z '' -o '' = 179d9a681d20
2024-05-08T01:35:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:35:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = present
2024-05-08T01:35:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror laggy
2024-05-08T01:35:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror laggy started
2024-05-08T01:35:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:35:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:35:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror laggy started
2024-05-08T01:35:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:35:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:35:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:35:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy
2024-05-08T01:35:43.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:35:43.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:35:43.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:43.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:43.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:35:43.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:35:43.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:43.459 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:35:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:35:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:35:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy
2024-05-08T01:35:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:35:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:35:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:35:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:35:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:35:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:35:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:35:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:35:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:35:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:35:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy
2024-05-08T01:35:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:35:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:35:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:35:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:35:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:35:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:35:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:35:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror laggy
2024-05-08T01:35:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:35:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:35:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:35:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror laggy
2024-05-08T01:35:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:35:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:35:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:35:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:35:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:35:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:35:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:35:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror laggy
2024-05-08T01:35:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:35:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n laggy ']'
2024-05-08T01:35:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/laggy'
2024-05-08T01:35:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy
2024-05-08T01:35:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:35:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:35:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:35:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:35:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:35:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:35:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:35:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy
2024-05-08T01:35:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:35:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:35:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:35:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:35:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:35:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:35:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:35:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:35:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:35:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:35:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:35:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:35:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:35:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:35:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:35:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy
2024-05-08T01:35:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror laggy
2024-05-08T01:35:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy ''
2024-05-08T01:35:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:35:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:35:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:35:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:35:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:35:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:35:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:35:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:35:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected]
2024-05-08T01:35:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected]
2024-05-08T01:35:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:44.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=3, tag_tid=3, entry_tid=3'
2024-05-08T01:35:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror laggy
2024-05-08T01:35:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy '..*'
2024-05-08T01:35:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:35:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:35:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:35:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:35:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:35:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:35:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:35:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:35:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected]
2024-05-08T01:35:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected]
2024-05-08T01:35:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=3, tag_tid=3, entry_tid=3'
2024-05-08T01:35:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=3, tag_tid=3, entry_tid=3' -a 'object_number=3, tag_tid=3, entry_tid=3' = 'object_number=3, tag_tid=3, entry_tid=3'
2024-05-08T01:35:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror laggy
2024-05-08T01:35:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy '..*'
2024-05-08T01:35:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:35:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:35:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:35:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:35:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:35:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:35:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:35:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:35:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected]
2024-05-08T01:35:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected]
2024-05-08T01:35:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=3, tag_tid=3, entry_tid=3'
2024-05-08T01:35:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror laggy
2024-05-08T01:35:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:35:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror-laggy.export
2024-05-08T01:35:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror-laggy.export
2024-05-08T01:35:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:35:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.export
2024-05-08T01:35:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror-laggy.export
2024-05-08T01:35:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror-laggy.export
2024-05-08T01:35:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:35:44.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror-laggy.export
2024-05-08T01:35:44.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.export /tmp/tmp.3xddryFtVF/cluster1-mirror-laggy.export
2024-05-08T01:35:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/laggy /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.export
2024-05-08T01:35:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:35:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/laggy /tmp/tmp.3xddryFtVF/cluster1-mirror-laggy.export
2024-05-08T01:35:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:35:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.export /tmp/tmp.3xddryFtVF/cluster1-mirror-laggy.export
2024-05-08T01:35:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.export /tmp/tmp.3xddryFtVF/cluster1-mirror-laggy.export
2024-05-08T01:35:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog ' - disconnected after max_concurrent_object_sets reached'
2024-05-08T01:35:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:35:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:35:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:35:44 - disconnected after max_concurrent_object_sets reached
2024-05-08T01:35:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:35:44 - disconnected after max_concurrent_object_sets reached
2024-05-08T01:35:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -z '' ']'
2024-05-08T01:35:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror stop mirror/laggy
2024-05-08T01:35:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:35:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:35:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:35:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:35:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:35:44.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:35:44.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:35:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror/laggy
2024-05-08T01:35:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:35:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:35:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:35:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:35:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:35:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:35:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:35:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:35:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:35:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:35:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:35:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:35:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:35:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:35:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:35:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror/laggy
2024-05-08T01:35:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror laggy
2024-05-08T01:35:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror laggy stopped
2024-05-08T01:35:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:35:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:35:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror laggy stopped
2024-05-08T01:35:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:35:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:35:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:35:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy
2024-05-08T01:35:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:35:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:35:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:35:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:35:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:35:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:35:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:35:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy
2024-05-08T01:35:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:35:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:35:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:35:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:35:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:35:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:35:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:35:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:35:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:35:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:35:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy
2024-05-08T01:35:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:35:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:35:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:35:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:35:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:35:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror laggy
2024-05-08T01:35:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy '..*'
2024-05-08T01:35:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:35:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:35:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:35:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:35:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:35:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 0
2024-05-08T01:35:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 0
2024-05-08T01:35:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:35:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected]
2024-05-08T01:35:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected]
2024-05-08T01:35:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=3, tag_tid=3, entry_tid=3'
2024-05-08T01:35:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_image_meta cluster2 mirror laggy conf_rbd_journal_max_concurrent_object_sets 1
2024-05-08T01:35:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:35:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local key=conf_rbd_journal_max_concurrent_object_sets
2024-05-08T01:35:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=1
2024-05-08T01:35:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 image-meta set mirror/laggy conf_rbd_journal_max_concurrent_object_sets 1
2024-05-08T01:35:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror laggy 20 16384
2024-05-08T01:35:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:35:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=20
2024-05-08T01:35:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=16384
2024-05-08T01:35:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 16384
2024-05-08T01:35:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/laggy --io-type write --io-size 16384 --io-threads 1 --io-total 327680 --io-pattern rand
2024-05-08T01:35:46.211 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 16384 io_threads 1 bytes 327680 pattern random
2024-05-08T01:35:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:35:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 20   ops/sec: 833.354   bytes/sec: 13 MiB/s
2024-05-08T01:35:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster2 mirror laggy 20 16384
2024-05-08T01:35:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:35:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:46.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=20
2024-05-08T01:35:46.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=16384
2024-05-08T01:35:46.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 16384
2024-05-08T01:35:46.246 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 bench mirror/laggy --io-type write --io-size 16384 --io-threads 1 --io-total 327680 --io-pattern rand
2024-05-08T01:35:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 16384 io_threads 1 bytes 327680 pattern random
2024-05-08T01:35:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:35:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 20   ops/sec: 555.569   bytes/sec: 8.7 MiB/s
2024-05-08T01:35:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror laggy
2024-05-08T01:35:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy '..*'
2024-05-08T01:35:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:35:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:35:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:35:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:35:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:35:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:35:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 1
2024-05-08T01:35:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:35:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:35:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=25, tag_tid=6, entry_tid=41], [object_number=16, tag_tid=6, entry_tid=40], [object_number=27, tag_tid=6, entry_tid=39], [object_number=18, tag_tid=6, entry_tid=38]]], state=connected]
2024-05-08T01:35:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=disconnected]
2024-05-08T01:35:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z ''
2024-05-08T01:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_image_meta cluster2 mirror laggy conf_rbd_journal_max_concurrent_object_sets 0
2024-05-08T01:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local key=conf_rbd_journal_max_concurrent_object_sets
2024-05-08T01:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=0
2024-05-08T01:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 image-meta set mirror/laggy conf_rbd_journal_max_concurrent_object_sets 0
2024-05-08T01:35:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog ' - replay is still stopped (disconnected) after restart'
2024-05-08T01:35:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:35:47.118 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:35:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:35:47 - replay is still stopped '(disconnected)' after restart
2024-05-08T01:35:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:35:47 - replay is still stopped (disconnected) after restart
2024-05-08T01:35:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror start mirror/laggy
2024-05-08T01:35:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:35:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:35:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:35:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:35:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:35:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:35:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:35:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/laggy
2024-05-08T01:35:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:35:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:35:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:35:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:35:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:35:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:35:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:35:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:35:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:35:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:35:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:35:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:35:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:35:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:35:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:35:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/laggy
2024-05-08T01:35:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror laggy
2024-05-08T01:35:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror laggy stopped
2024-05-08T01:35:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:35:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:35:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror laggy stopped
2024-05-08T01:35:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:35:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:35:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:35:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy
2024-05-08T01:35:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:35:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:35:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:35:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:35:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:35:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:35:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:35:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:35:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy
2024-05-08T01:35:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:35:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:35:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:35:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:35:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:35:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:35:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:35:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:35:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:35:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:35:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:35:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:35:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:35:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:35:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:35:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:35:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:35:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:35:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:35:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:35:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:35:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:35:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:35:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:35:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:35:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:35:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:35:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:35:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy
2024-05-08T01:35:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:35:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:35:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:35:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:35:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:35:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror laggy up+error disconnected
2024-05-08T01:35:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+error
2024-05-08T01:35:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=disconnected
2024-05-08T01:35:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:35:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:35:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror laggy up+error disconnected ''
2024-05-08T01:35:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+error
2024-05-08T01:35:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=disconnected
2024-05-08T01:35:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:35:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:35:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:35:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/laggy
2024-05-08T01:35:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='laggy:
2024-05-08T01:35:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   5ddea2ca-740c-4e60-ac31-ee957a3be11f
2024-05-08T01:35:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+error
2024-05-08T01:35:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: disconnected
2024-05-08T01:35:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:35:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:35:47
2024-05-08T01:35:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:35:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:35:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: down+unknown
2024-05-08T01:35:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: status not found
2024-05-08T01:35:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: '
2024-05-08T01:35:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+error'
2024-05-08T01:35:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+error
2024-05-08T01:35:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*disconnected'
2024-05-08T01:35:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: disconnected
2024-05-08T01:35:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:35:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+error
2024-05-08T01:35:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:35:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+error
2024-05-08T01:35:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:35:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:35:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:35:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:35:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:35:47'
2024-05-08T01:35:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror laggy up+error '2024-05-08 01:35:47'
2024-05-08T01:35:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+error
2024-05-08T01:35:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:35:47'
2024-05-08T01:35:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:35:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:35:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:35:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>6038</service_id><instance_id>6045</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>6042</service_id><instance_id>6055</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>6049</service_id><instance_id>6059</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>6056</service_id><instance_id>6062</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>laggy</name><global_id>5ddea2ca-740c-4e60-ac31-ee957a3be11f</global_id><state>up+error</state><description>disconnected</description><daemon_service><service_id>6038</service_id><instance_id>6045</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:35:47</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>down+unknown</state><description>status not found</description><last_update></last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:35:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:35:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/last_update'
2024-05-08T01:35:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:35:47'
2024-05-08T01:35:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/state'
2024-05-08T01:35:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+error
2024-05-08T01:35:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+error
2024-05-08T01:35:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+error
2024-05-08T01:35:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+error
2024-05-08T01:35:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog ' - replay started after resync requested'
2024-05-08T01:35:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:35:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:35:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:35:49 - replay started after resync requested
2024-05-08T01:35:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:35:49 - replay started after resync requested
2024-05-08T01:35:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ request_resync_image cluster1 mirror laggy image_id
2024-05-08T01:35:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id_var_name=image_id
2024-05-08T01:35:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:35:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:35:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval 'image_id='\''179d9a681d20'\'''
2024-05-08T01:35:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ image_id=179d9a681d20
2024-05-08T01:35:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval 'test -n "$image_id"'
2024-05-08T01:35:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -n 179d9a681d20
2024-05-08T01:35:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/laggy
2024-05-08T01:35:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Flagged image for resync from primary
2024-05-08T01:35:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror laggy deleted 179d9a681d20
2024-05-08T01:35:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=deleted
2024-05-08T01:35:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=179d9a681d20
2024-05-08T01:35:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:35:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 179d9a681d20
2024-05-08T01:35:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:35:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror laggy deleted 179d9a681d20
2024-05-08T01:35:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:35:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=179d9a681d20
2024-05-08T01:35:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:35:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:35:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=179d9a681d20
2024-05-08T01:35:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 179d9a681d20
2024-05-08T01:35:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z 179d9a681d20 -o 179d9a681d20 = 179d9a681d20
2024-05-08T01:35:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:35:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = present
2024-05-08T01:35:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror laggy deleted 179d9a681d20
2024-05-08T01:35:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:35:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=179d9a681d20
2024-05-08T01:35:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:35:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:35:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=179d9a681d20
2024-05-08T01:35:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 179d9a681d20
2024-05-08T01:35:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z 179d9a681d20 -o 179d9a681d20 = 179d9a681d20
2024-05-08T01:35:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:35:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = present
2024-05-08T01:35:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:35:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror laggy deleted 179d9a681d20
2024-05-08T01:35:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:35:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=179d9a681d20
2024-05-08T01:35:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:35:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:35:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=179d9a681d20
2024-05-08T01:35:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 179d9a681d20
2024-05-08T01:35:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z 179d9a681d20 -o 179d9a681d20 = 179d9a681d20
2024-05-08T01:35:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:35:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = present
2024-05-08T01:35:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 4
2024-05-08T01:35:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror laggy deleted 179d9a681d20
2024-05-08T01:35:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:35:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=179d9a681d20
2024-05-08T01:35:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:35:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:35:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image laggy: (2) No such file or directory
2024-05-08T01:35:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:35:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = deleted
2024-05-08T01:35:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:35:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror laggy present
2024-05-08T01:35:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=present
2024-05-08T01:35:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:35:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:35:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:35:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:35:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image laggy: (2) No such file or directory
2024-05-08T01:35:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=
2024-05-08T01:35:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:35:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror laggy present ''
2024-05-08T01:35:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:35:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:35:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:35:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:35:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image laggy: (2) No such file or directory
2024-05-08T01:35:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:35:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:35:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:35:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror laggy present ''
2024-05-08T01:35:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:35:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:35:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:35:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:35:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:35:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:35:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:35:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:35:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:35:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:35:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:35:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:35:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:35:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image laggy: (2) No such file or directory
2024-05-08T01:35:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:35:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:35:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:35:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:35:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:36:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror laggy present ''
2024-05-08T01:36:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:36:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:36:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:36:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:36:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:36:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:36:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:36:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=179d4512816a
2024-05-08T01:36:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 179d4512816a
2024-05-08T01:36:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z '' -o '' = 179d4512816a
2024-05-08T01:36:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:36:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = present
2024-05-08T01:36:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror laggy
2024-05-08T01:36:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror laggy started
2024-05-08T01:36:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:36:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:36:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:36:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror laggy started
2024-05-08T01:36:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:36:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:36:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:36:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy
2024-05-08T01:36:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:36:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:36:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:36:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:36:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:36:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:36:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:36:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:36:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy
2024-05-08T01:36:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:36:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:36:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:36:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:36:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:36:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:36:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:36:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:36:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:36:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:36:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:36:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:36:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:36:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:36:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:36:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:36:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:36:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:36:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:36:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:36:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:36:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:36:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy
2024-05-08T01:36:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:36:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:36:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:36:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:36:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:36:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:36:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:36:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror laggy
2024-05-08T01:36:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:36:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:36:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror laggy
2024-05-08T01:36:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:36:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:36:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:36:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:36:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:36:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:36:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror laggy
2024-05-08T01:36:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:36:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n laggy ']'
2024-05-08T01:36:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/laggy'
2024-05-08T01:36:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy
2024-05-08T01:36:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:36:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:36:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:36:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:36:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:36:02.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:36:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy
2024-05-08T01:36:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:36:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:36:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:36:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:36:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:36:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:36:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:36:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:36:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:36:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:36:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:36:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:36:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:36:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:36:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:36:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:36:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:36:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:36:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:36:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy
2024-05-08T01:36:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror laggy
2024-05-08T01:36:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy ''
2024-05-08T01:36:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:36:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:36:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:36:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:36:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 4
2024-05-08T01:36:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:36:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:36:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected]
2024-05-08T01:36:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0], [object_number=27, tag_tid=6, entry_tid=39], [object_number=18, tag_tid=6, entry_tid=38]]], state=connected]
2024-05-08T01:36:02.241 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=27, tag_tid=8, entry_tid=3'
2024-05-08T01:36:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror laggy
2024-05-08T01:36:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy '..*'
2024-05-08T01:36:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:36:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:36:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:36:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:36:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 4
2024-05-08T01:36:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:36:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:36:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected]
2024-05-08T01:36:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0], [object_number=27, tag_tid=6, entry_tid=39], [object_number=18, tag_tid=6, entry_tid=38]]], state=connected]
2024-05-08T01:36:02.507 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=25, tag_tid=8, entry_tid=1'
2024-05-08T01:36:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=27, tag_tid=8, entry_tid=3' -a 'object_number=27, tag_tid=8, entry_tid=3' = 'object_number=25, tag_tid=8, entry_tid=1'
2024-05-08T01:36:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=25, tag_tid=8, entry_tid=1' '!=' ''
2024-05-08T01:36:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:36:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=25, tag_tid=8, entry_tid=1' = ''
2024-05-08T01:36:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=25, tag_tid=8, entry_tid=1'
2024-05-08T01:36:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=27, tag_tid=8, entry_tid=3'
2024-05-08T01:36:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:36:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:36:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=8
2024-05-08T01:36:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=25, tag_tid=8, entry_tid=1'
2024-05-08T01:36:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:36:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:36:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=8
2024-05-08T01:36:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=27, tag_tid=8, entry_tid=3'
2024-05-08T01:36:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:36:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:36:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=3
2024-05-08T01:36:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=25, tag_tid=8, entry_tid=1'
2024-05-08T01:36:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:36:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:36:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=1
2024-05-08T01:36:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 8 = 8 -a 3 -le 1
2024-05-08T01:36:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:36:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:36:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:36:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror laggy
2024-05-08T01:36:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:36:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n laggy ']'
2024-05-08T01:36:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/laggy'
2024-05-08T01:36:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy
2024-05-08T01:36:02.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:36:02.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:02.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:02.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:36:02.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:36:02.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:02.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:36:02.722 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:36:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:36:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy
2024-05-08T01:36:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:36:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:36:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:36:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:36:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:36:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:36:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:36:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:36:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:36:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:36:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:36:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:36:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:36:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:36:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:36:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:36:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:36:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:36:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:36:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy
2024-05-08T01:36:02.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror laggy
2024-05-08T01:36:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy ''
2024-05-08T01:36:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:36:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:36:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:36:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:36:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 6
2024-05-08T01:36:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:36:02.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:36:02.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected]
2024-05-08T01:36:02.943 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected]
2024-05-08T01:36:02.946 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=27, tag_tid=8, entry_tid=3'
2024-05-08T01:36:02.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror laggy
2024-05-08T01:36:02.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy '..*'
2024-05-08T01:36:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:36:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:36:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:36:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:36:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 6
2024-05-08T01:36:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:36:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:36:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected]
2024-05-08T01:36:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected]
2024-05-08T01:36:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=27, tag_tid=8, entry_tid=3'
2024-05-08T01:36:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=27, tag_tid=8, entry_tid=3' -a 'object_number=27, tag_tid=8, entry_tid=3' = 'object_number=27, tag_tid=8, entry_tid=3'
2024-05-08T01:36:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror laggy
2024-05-08T01:36:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy '..*'
2024-05-08T01:36:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:36:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:36:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:36:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:36:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:03.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 6
2024-05-08T01:36:03.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:36:03.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:36:03.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected]
2024-05-08T01:36:03.049 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected]
2024-05-08T01:36:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=27, tag_tid=8, entry_tid=3'
2024-05-08T01:36:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror laggy
2024-05-08T01:36:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:36:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror-laggy.export
2024-05-08T01:36:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror-laggy.export
2024-05-08T01:36:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.export
2024-05-08T01:36:03.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror-laggy.export
2024-05-08T01:36:03.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror-laggy.export
2024-05-08T01:36:03.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror-laggy.export
2024-05-08T01:36:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.export /tmp/tmp.3xddryFtVF/cluster1-mirror-laggy.export
2024-05-08T01:36:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/laggy /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.export
2024-05-08T01:36:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:36:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/laggy /tmp/tmp.3xddryFtVF/cluster1-mirror-laggy.export
2024-05-08T01:36:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:36:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.export /tmp/tmp.3xddryFtVF/cluster1-mirror-laggy.export
2024-05-08T01:36:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.export /tmp/tmp.3xddryFtVF/cluster1-mirror-laggy.export
2024-05-08T01:36:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog ' - rbd_mirroring_resync_after_disconnect config option'
2024-05-08T01:36:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:36:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:36:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:36:03 - rbd_mirroring_resync_after_disconnect config option
2024-05-08T01:36:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:36:03 - rbd_mirroring_resync_after_disconnect config option
2024-05-08T01:36:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_image_meta cluster2 mirror laggy conf_rbd_mirroring_resync_after_disconnect true
2024-05-08T01:36:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local key=conf_rbd_mirroring_resync_after_disconnect
2024-05-08T01:36:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=true
2024-05-08T01:36:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 image-meta set mirror/laggy conf_rbd_mirroring_resync_after_disconnect true
2024-05-08T01:36:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror laggy
2024-05-08T01:36:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:36:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:36:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror laggy
2024-05-08T01:36:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:36:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:36:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:36:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:36:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:36:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:36:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror laggy
2024-05-08T01:36:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:36:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n laggy ']'
2024-05-08T01:36:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/laggy'
2024-05-08T01:36:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy
2024-05-08T01:36:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:36:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:36:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:36:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:36:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:36:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:36:03.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:03.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy
2024-05-08T01:36:03.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:36:03.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:03.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:36:03.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:36:03.744 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:36:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:36:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:36:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:36:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:36:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:36:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:36:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:36:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:36:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:36:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:36:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:36:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:36:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:36:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:36:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:36:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy
2024-05-08T01:36:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror laggy
2024-05-08T01:36:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy ''
2024-05-08T01:36:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:36:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:36:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:36:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:36:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 6
2024-05-08T01:36:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:36:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:36:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected]
2024-05-08T01:36:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected]
2024-05-08T01:36:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=25, tag_tid=9, entry_tid=1'
2024-05-08T01:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror laggy
2024-05-08T01:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy '..*'
2024-05-08T01:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:36:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 6
2024-05-08T01:36:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:36:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:36:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected]
2024-05-08T01:36:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected]
2024-05-08T01:36:04.012 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=27, tag_tid=8, entry_tid=3'
2024-05-08T01:36:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=25, tag_tid=9, entry_tid=1' -a 'object_number=25, tag_tid=9, entry_tid=1' = 'object_number=27, tag_tid=8, entry_tid=3'
2024-05-08T01:36:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=27, tag_tid=8, entry_tid=3' '!=' ''
2024-05-08T01:36:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:36:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=27, tag_tid=8, entry_tid=3' = ''
2024-05-08T01:36:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=27, tag_tid=8, entry_tid=3'
2024-05-08T01:36:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=25, tag_tid=9, entry_tid=1'
2024-05-08T01:36:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:36:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:36:04.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=9
2024-05-08T01:36:04.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=27, tag_tid=8, entry_tid=3'
2024-05-08T01:36:04.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:36:04.016 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:36:04.017 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=8
2024-05-08T01:36:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=25, tag_tid=9, entry_tid=1'
2024-05-08T01:36:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:36:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:36:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=1
2024-05-08T01:36:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=27, tag_tid=8, entry_tid=3'
2024-05-08T01:36:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:36:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:36:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=3
2024-05-08T01:36:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 9 = 8 -a 1 -le 3
2024-05-08T01:36:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:36:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:36:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:36:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror laggy
2024-05-08T01:36:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:36:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n laggy ']'
2024-05-08T01:36:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/laggy'
2024-05-08T01:36:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy
2024-05-08T01:36:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:36:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:36:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:36:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:36:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:36:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:36:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy
2024-05-08T01:36:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:36:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:36:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:36:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:36:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:36:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:36:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:36:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:36:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:36:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:36:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:36:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:36:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:36:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:36:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:36:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:36:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:36:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:36:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:36:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy
2024-05-08T01:36:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror laggy
2024-05-08T01:36:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy ''
2024-05-08T01:36:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:36:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:36:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:36:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:36:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 6
2024-05-08T01:36:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:36:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:36:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected]
2024-05-08T01:36:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected]
2024-05-08T01:36:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=25, tag_tid=9, entry_tid=1'
2024-05-08T01:36:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror laggy
2024-05-08T01:36:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy '..*'
2024-05-08T01:36:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:36:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:36:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:36:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:36:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 6
2024-05-08T01:36:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:36:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:36:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected]
2024-05-08T01:36:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected]
2024-05-08T01:36:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=27, tag_tid=8, entry_tid=3'
2024-05-08T01:36:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=25, tag_tid=9, entry_tid=1' -a 'object_number=25, tag_tid=9, entry_tid=1' = 'object_number=27, tag_tid=8, entry_tid=3'
2024-05-08T01:36:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=27, tag_tid=8, entry_tid=3' '!=' 'object_number=27, tag_tid=8, entry_tid=3'
2024-05-08T01:36:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:36:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.4
2024-05-08T01:36:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror laggy
2024-05-08T01:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n laggy ']'
2024-05-08T01:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/laggy'
2024-05-08T01:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy
2024-05-08T01:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:36:04.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:36:04.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:36:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy
2024-05-08T01:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:36:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:36:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:36:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:36:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:36:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:36:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:36:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:36:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:36:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:36:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy
2024-05-08T01:36:05.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:05.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror laggy
2024-05-08T01:36:05.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:05.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:05.043 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy ''
2024-05-08T01:36:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:36:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:36:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:36:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:36:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 6
2024-05-08T01:36:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:36:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:36:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected]
2024-05-08T01:36:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected]
2024-05-08T01:36:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=25, tag_tid=9, entry_tid=1'
2024-05-08T01:36:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror laggy
2024-05-08T01:36:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy '..*'
2024-05-08T01:36:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:36:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:36:05.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:36:05.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:05.103 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:36:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 6
2024-05-08T01:36:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:36:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:36:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected]
2024-05-08T01:36:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected]
2024-05-08T01:36:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=25, tag_tid=9, entry_tid=1'
2024-05-08T01:36:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=25, tag_tid=9, entry_tid=1' -a 'object_number=25, tag_tid=9, entry_tid=1' = 'object_number=25, tag_tid=9, entry_tid=1'
2024-05-08T01:36:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:36:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:36:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:36:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=179d4512816a
2024-05-08T01:36:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ disconnect_image cluster2 mirror laggy
2024-05-08T01:36:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 journal client disconnect --image mirror/laggy
2024-05-08T01:36:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:client d8559015-eaca-4ba8-a079-91f9b3432357 disconnected
2024-05-08T01:36:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror laggy deleted 179d4512816a
2024-05-08T01:36:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=deleted
2024-05-08T01:36:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=179d4512816a
2024-05-08T01:36:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:36:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 179d4512816a
2024-05-08T01:36:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:36:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:36:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror laggy deleted 179d4512816a
2024-05-08T01:36:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=deleted
2024-05-08T01:36:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=179d4512816a
2024-05-08T01:36:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:36:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:36:05.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:36:05.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:05.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:05.369 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:05.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:36:05.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:36:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image laggy: (2) No such file or directory
2024-05-08T01:36:05.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:36:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:36:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test deleted = deleted
2024-05-08T01:36:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_present cluster1 mirror laggy present
2024-05-08T01:36:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=present
2024-05-08T01:36:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:36:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:36:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:36:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:36:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:36:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:36:05.467 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image laggy: (2) No such file or directory
2024-05-08T01:36:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image_id=
2024-05-08T01:36:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:36:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.1
2024-05-08T01:36:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror laggy present ''
2024-05-08T01:36:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:36:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:36:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:36:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:36:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:36:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:36:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:36:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image laggy: (2) No such file or directory
2024-05-08T01:36:05.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:36:05.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:36:05.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:36:05.623 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:36:05.624 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:36:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror laggy present ''
2024-05-08T01:36:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:36:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:36:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:36:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:36:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:36:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:36:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:36:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image laggy: (2) No such file or directory
2024-05-08T01:36:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:36:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:36:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:36:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:36:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:36:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror laggy present ''
2024-05-08T01:36:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:36:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:36:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:36:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:36:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:36:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:36:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:36:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image laggy: (2) No such file or directory
2024-05-08T01:36:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=
2024-05-08T01:36:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:36:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = deleted
2024-05-08T01:36:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32
2024-05-08T01:36:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 4
2024-05-08T01:36:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_present cluster1 mirror laggy present ''
2024-05-08T01:36:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=present
2024-05-08T01:36:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id=
2024-05-08T01:36:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=deleted
2024-05-08T01:36:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_image_id
2024-05-08T01:36:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror laggy
2024-05-08T01:36:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/laggy
2024-05-08T01:36:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:36:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_image_id=179d50100845
2024-05-08T01:36:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 179d50100845
2024-05-08T01:36:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z '' -o '' = 179d50100845
2024-05-08T01:36:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=present
2024-05-08T01:36:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test present = present
2024-05-08T01:36:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_started cluster1 mirror laggy
2024-05-08T01:36:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror laggy started
2024-05-08T01:36:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=started
2024-05-08T01:36:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:36:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:36:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror laggy started
2024-05-08T01:36:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=started
2024-05-08T01:36:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:36:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:36:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy
2024-05-08T01:36:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:36:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:36:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:36:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:36:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:36:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:36:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:36:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:36:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy
2024-05-08T01:36:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:36:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:36:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:36:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:36:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:36:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:36:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:36:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:36:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:36:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:36:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:36:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:36:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:36:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:36:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:36:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:36:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:36:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:36:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:36:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:36:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:36:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:36:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy
2024-05-08T01:36:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:36:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Replaying"'
2024-05-08T01:36:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:36:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Replaying"'
2024-05-08T01:36:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:    "state": "Replaying"
2024-05-08T01:36:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ current_state=started
2024-05-08T01:36:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test started = started
2024-05-08T01:36:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror laggy
2024-05-08T01:36:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:36:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:36:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror laggy
2024-05-08T01:36:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:36:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:36:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:36:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:36:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:36:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:36:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror laggy
2024-05-08T01:36:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:36:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n laggy ']'
2024-05-08T01:36:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/laggy'
2024-05-08T01:36:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy
2024-05-08T01:36:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:36:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:36:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:36:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:36:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:36:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:36:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy
2024-05-08T01:36:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:36:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:36:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:36:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:36:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:36:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:36:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:36:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:36:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:36:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:36:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:36:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:36:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:36:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:36:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:36:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:36:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:36:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:36:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:36:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy
2024-05-08T01:36:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror laggy
2024-05-08T01:36:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy ''
2024-05-08T01:36:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:36:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:36:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:36:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:36:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 6
2024-05-08T01:36:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:36:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:36:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected]
2024-05-08T01:36:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected]
2024-05-08T01:36:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=27, tag_tid=10, entry_tid=3'
2024-05-08T01:36:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror laggy
2024-05-08T01:36:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy '..*'
2024-05-08T01:36:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:36:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:36:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:36:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:36:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 6
2024-05-08T01:36:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:36:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:36:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected]
2024-05-08T01:36:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected]
2024-05-08T01:36:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=27, tag_tid=10, entry_tid=3'
2024-05-08T01:36:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=27, tag_tid=10, entry_tid=3' -a 'object_number=27, tag_tid=10, entry_tid=3' = 'object_number=27, tag_tid=10, entry_tid=3'
2024-05-08T01:36:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror laggy
2024-05-08T01:36:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy '..*'
2024-05-08T01:36:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:36:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:36:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:36:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:14.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:14.429 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:36:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 6
2024-05-08T01:36:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:36:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:36:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected]
2024-05-08T01:36:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected]
2024-05-08T01:36:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=27, tag_tid=10, entry_tid=3'
2024-05-08T01:36:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ compare_images mirror laggy
2024-05-08T01:36:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local ret=0
2024-05-08T01:36:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster2-mirror-laggy.export
2024-05-08T01:36:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster2-mirror-laggy.export
2024-05-08T01:36:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local rmt_export=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.export
2024-05-08T01:36:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ mkfname cluster1-mirror-laggy.export
2024-05-08T01:36:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo cluster1-mirror-laggy.export
2024-05-08T01:36:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:14.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loc_export=/tmp/tmp.3xddryFtVF/cluster1-mirror-laggy.export
2024-05-08T01:36:14.483 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.export /tmp/tmp.3xddryFtVF/cluster1-mirror-laggy.export
2024-05-08T01:36:14.484 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 export mirror/laggy /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.export
2024-05-08T01:36:14.596 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:36:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 export mirror/laggy /tmp/tmp.3xddryFtVF/cluster1-mirror-laggy.export
2024-05-08T01:36:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Exporting image: 3% complete...
Exporting image: 6% complete...
Exporting image: 9% complete...
Exporting image: 12% complete...
Exporting image: 15% complete...
Exporting image: 18% complete...
Exporting image: 21% complete...
Exporting image: 25% complete...
Exporting image: 28% complete...
Exporting image: 31% complete...
Exporting image: 34% complete...
Exporting image: 37% complete...
Exporting image: 40% complete...
Exporting image: 43% complete...
Exporting image: 46% complete...
Exporting image: 50% complete...
Exporting image: 53% complete...
Exporting image: 56% complete...
Exporting image: 59% complete...
Exporting image: 62% complete...
Exporting image: 65% complete...
Exporting image: 68% complete...
Exporting image: 71% complete...
Exporting image: 75% complete...
Exporting image: 78% complete...
Exporting image: 81% complete...
Exporting image: 84% complete...
Exporting image: 87% complete...
Exporting image: 90% complete...
Exporting image: 93% complete...
Exporting image: 96% complete...
Exporting image: 100% complete...done.
2024-05-08T01:36:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmp /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.export /tmp/tmp.3xddryFtVF/cluster1-mirror-laggy.export
2024-05-08T01:36:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.export /tmp/tmp.3xddryFtVF/cluster1-mirror-laggy.export
2024-05-08T01:36:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_image_meta cluster2 mirror laggy conf_rbd_mirroring_resync_after_disconnect false
2024-05-08T01:36:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local key=conf_rbd_mirroring_resync_after_disconnect
2024-05-08T01:36:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=false
2024-05-08T01:36:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 image-meta set mirror/laggy conf_rbd_mirroring_resync_after_disconnect false
2024-05-08T01:36:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror laggy
2024-05-08T01:36:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:36:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:36:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror laggy
2024-05-08T01:36:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local local_cluster=cluster1
2024-05-08T01:36:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s master_pos mirror_pos last_mirror_pos
2024-05-08T01:36:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local master_tag master_entry mirror_tag mirror_entry
2024-05-08T01:36:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:36:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:36:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:36:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror laggy
2024-05-08T01:36:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:36:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n laggy ']'
2024-05-08T01:36:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/laggy'
2024-05-08T01:36:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy
2024-05-08T01:36:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:36:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:36:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:36:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:36:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:36:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:36:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy
2024-05-08T01:36:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:36:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:36:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:36:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:36:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:36:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:36:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:36:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:36:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:36:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:36:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy
2024-05-08T01:36:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror laggy
2024-05-08T01:36:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy ''
2024-05-08T01:36:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:36:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:36:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:36:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:36:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 6
2024-05-08T01:36:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:36:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:36:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected]
2024-05-08T01:36:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected]
2024-05-08T01:36:15.343 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:15.345 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=25, tag_tid=11, entry_tid=1'
2024-05-08T01:36:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror laggy
2024-05-08T01:36:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy '..*'
2024-05-08T01:36:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:36:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:36:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:36:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:36:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:15.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 6
2024-05-08T01:36:15.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:36:15.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:36:15.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected]
2024-05-08T01:36:15.394 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected]
2024-05-08T01:36:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=27, tag_tid=10, entry_tid=3'
2024-05-08T01:36:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=25, tag_tid=11, entry_tid=1' -a 'object_number=25, tag_tid=11, entry_tid=1' = 'object_number=27, tag_tid=10, entry_tid=3'
2024-05-08T01:36:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=27, tag_tid=10, entry_tid=3' '!=' ''
2024-05-08T01:36:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:36:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 'object_number=27, tag_tid=10, entry_tid=3' = ''
2024-05-08T01:36:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_mirror_pos='object_number=27, tag_tid=10, entry_tid=3'
2024-05-08T01:36:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=25, tag_tid=11, entry_tid=1'
2024-05-08T01:36:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:36:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:36:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_tag=11
2024-05-08T01:36:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=27, tag_tid=10, entry_tid=3'
2024-05-08T01:36:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'tag_tid=[0-9]*'
2024-05-08T01:36:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:36:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_tag=10
2024-05-08T01:36:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=25, tag_tid=11, entry_tid=1'
2024-05-08T01:36:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:36:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:36:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_entry=1
2024-05-08T01:36:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo 'object_number=27, tag_tid=10, entry_tid=3'
2024-05-08T01:36:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -Eo 'entry_tid=[0-9]*'
2024-05-08T01:36:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cut -d= -f 2
2024-05-08T01:36:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_entry=3
2024-05-08T01:36:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 11 = 10 -a 1 -le 3
2024-05-08T01:36:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ true
2024-05-08T01:36:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32
2024-05-08T01:36:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0.2
2024-05-08T01:36:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ flush cluster1 mirror laggy
2024-05-08T01:36:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'cmd=rbd mirror flush'
2024-05-08T01:36:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n laggy ']'
2024-05-08T01:36:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cmd='rbd mirror flush mirror/laggy'
2024-05-08T01:36:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy
2024-05-08T01:36:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_instance=cluster1
2024-05-08T01:36:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance=cluster1
2024-05-08T01:36:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local loop_instance
2024-05-08T01:36:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:36:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:36:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:36:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy
2024-05-08T01:36:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:36:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:36:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:36:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster1:0
2024-05-08T01:36:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:36:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:36:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster1
2024-05-08T01:36:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:36:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster1
2024-05-08T01:36:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:36:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:36:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:36:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:36:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:36:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:36:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:36:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:36:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:36:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:36:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy
2024-05-08T01:36:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_master_journal_position cluster2 mirror laggy
2024-05-08T01:36:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy ''
2024-05-08T01:36:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local id_regexp=
2024-05-08T01:36:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:36:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:36:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:36:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 6
2024-05-08T01:36:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:36:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:36:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected]
2024-05-08T01:36:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected]
2024-05-08T01:36:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ master_pos='object_number=25, tag_tid=11, entry_tid=1'
2024-05-08T01:36:15.830 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror laggy
2024-05-08T01:36:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy '..*'
2024-05-08T01:36:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:36:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:36:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:36:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:36:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 6
2024-05-08T01:36:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:36:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:36:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected]
2024-05-08T01:36:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected]
2024-05-08T01:36:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ mirror_pos='object_number=25, tag_tid=11, entry_tid=1'
2024-05-08T01:36:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n 'object_number=25, tag_tid=11, entry_tid=1' -a 'object_number=25, tag_tid=11, entry_tid=1' = 'object_number=25, tag_tid=11, entry_tid=1'
2024-05-08T01:36:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ disconnect_image cluster2 mirror laggy
2024-05-08T01:36:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 journal client disconnect --image mirror/laggy
2024-05-08T01:36:15.931 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:client d8559015-eaca-4ba8-a079-91f9b3432357 disconnected
2024-05-08T01:36:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_mirror_journal_position cluster2 mirror laggy
2024-05-08T01:36:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_journal_position cluster2 mirror laggy '..*'
2024-05-08T01:36:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=laggy
2024-05-08T01:36:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local 'id_regexp=..*'
2024-05-08T01:36:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ mkfname cluster2-mirror-laggy.status
2024-05-08T01:36:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo cluster2-mirror-laggy.status
2024-05-08T01:36:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ sed -e 's|[/ ]|_|g'
2024-05-08T01:36:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local status_log=/tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy
2024-05-08T01:36:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ tee /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:minimum_set: 6
2024-05-08T01:36:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:active_set: 6
2024-05-08T01:36:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:registered clients:
2024-05-08T01:36:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected]
2024-05-08T01:36:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:	[id=d8559015-eaca-4ba8-a079-91f9b3432357, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=disconnected]
2024-05-08T01:36:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.3xddryFtVF/cluster2-mirror-laggy.status
2024-05-08T01:36:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -z ''
2024-05-08T01:36:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror laggy
2024-05-08T01:36:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror laggy stopped
2024-05-08T01:36:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:36:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:36:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:36:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror laggy stopped
2024-05-08T01:36:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:36:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:36:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:36:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy
2024-05-08T01:36:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:36:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:36:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:36:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:36:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:36:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:36:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:36:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:36:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy
2024-05-08T01:36:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:36:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:36:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:36:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:36:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:36:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:36:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:36:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:36:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:36:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:36:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:36:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:36:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:36:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:36:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:36:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:36:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:36:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:36:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:36:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:36:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:36:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:36:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy
2024-05-08T01:36:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:36:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:36:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:36:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:36:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:36:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror laggy up+error disconnected
2024-05-08T01:36:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+error
2024-05-08T01:36:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=disconnected
2024-05-08T01:36:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:36:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:36:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror laggy up+error disconnected ''
2024-05-08T01:36:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+error
2024-05-08T01:36:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=disconnected
2024-05-08T01:36:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:36:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:36:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:36:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/laggy
2024-05-08T01:36:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='laggy:
2024-05-08T01:36:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   5ddea2ca-740c-4e60-ac31-ee957a3be11f
2024-05-08T01:36:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+error
2024-05-08T01:36:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: disconnected
2024-05-08T01:36:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:36:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:36:15
2024-05-08T01:36:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:36:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:36:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: down+unknown
2024-05-08T01:36:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: status not found
2024-05-08T01:36:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: '
2024-05-08T01:36:18.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+error'
2024-05-08T01:36:18.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+error
2024-05-08T01:36:18.229 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*disconnected'
2024-05-08T01:36:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: disconnected
2024-05-08T01:36:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:36:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+error
2024-05-08T01:36:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:36:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+error
2024-05-08T01:36:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:36:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:36:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:36:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:36:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:36:15'
2024-05-08T01:36:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror laggy up+error '2024-05-08 01:36:15'
2024-05-08T01:36:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+error
2024-05-08T01:36:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:36:15'
2024-05-08T01:36:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:36:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:36:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:36:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>6038</service_id><instance_id>6045</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>6042</service_id><instance_id>6055</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>6049</service_id><instance_id>6059</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>6056</service_id><instance_id>6062</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>laggy</name><global_id>5ddea2ca-740c-4e60-ac31-ee957a3be11f</global_id><state>up+error</state><description>disconnected</description><daemon_service><service_id>6038</service_id><instance_id>6045</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:36:15</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>down+unknown</state><description>status not found</description><last_update></last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:36:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:36:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/last_update'
2024-05-08T01:36:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:36:15'
2024-05-08T01:36:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/state'
2024-05-08T01:36:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+error
2024-05-08T01:36:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+error
2024-05-08T01:36:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+error
2024-05-08T01:36:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+error
2024-05-08T01:36:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster2 mirror laggy
2024-05-08T01:36:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:36:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:36:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster2 mirror laggy
2024-05-08T01:36:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=laggy
2024-05-08T01:36:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 snap purge mirror/laggy
2024-05-08T01:36:18.374 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rm mirror/laggy
2024-05-08T01:36:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:36:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: split-brain'
2024-05-08T01:36:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:36:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:36:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:36:18 TEST: split-brain
2024-05-08T01:36:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:36:18 TEST: split-brain
2024-05-08T01:36:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ image=split-brain
2024-05-08T01:36:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image_and_enable_mirror cluster2 mirror split-brain journal
2024-05-08T01:36:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local mode=journal
2024-05-08T01:36:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n journal ']'
2024-05-08T01:36:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ create_image cluster2 mirror split-brain
2024-05-08T01:36:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ shift
2024-05-08T01:36:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=128
2024-05-08T01:36:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:36:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/split-brain
2024-05-08T01:36:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ is_pool_mirror_mode_image mirror
2024-05-08T01:36:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' pool = image ']'
2024-05-08T01:36:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ case "${pool}" in
2024-05-08T01:36:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:36:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror split-brain
2024-05-08T01:36:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:36:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position
2024-05-08T01:36:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:36:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:36:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:36:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:36:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position ''
2024-05-08T01:36:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:36:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:36:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:36:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:36:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:36:19.555 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain
2024-05-08T01:36:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image split-brain: (2) No such file or directory
2024-05-08T01:36:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status=
2024-05-08T01:36:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:36:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:36:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:36:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position ''
2024-05-08T01:36:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:36:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:36:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:36:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:36:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:36:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain
2024-05-08T01:36:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='split-brain:
2024-05-08T01:36:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   f4adcb09-1494-4073-a119-240379e5f7c0
2024-05-08T01:36:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       down+unknown
2024-05-08T01:36:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: status not found
2024-05-08T01:36:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update:
2024-05-08T01:36:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:36:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:36:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: down+unknown
2024-05-08T01:36:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: status not found
2024-05-08T01:36:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: '
2024-05-08T01:36:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:36:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:36:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:21.692 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 4
2024-05-08T01:36:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position ''
2024-05-08T01:36:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:36:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:36:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:36:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:36:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:36:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain
2024-05-08T01:36:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='split-brain:
2024-05-08T01:36:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   f4adcb09-1494-4073-a119-240379e5f7c0
2024-05-08T01:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}
2024-05-08T01:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:36:21
2024-05-08T01:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: down+unknown
2024-05-08T01:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: status not found
2024-05-08T01:36:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: '
2024-05-08T01:36:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:36:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:36:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:36:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}
2024-05-08T01:36:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:36:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:36:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:36:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:36:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:36:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:36:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:36:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:36:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:36:21'
2024-05-08T01:36:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+replaying '2024-05-08 01:36:21'
2024-05-08T01:36:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:36:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:36:21'
2024-05-08T01:36:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:36:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:36:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:36:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>6038</service_id><instance_id>6045</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>6042</service_id><instance_id>6055</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>6049</service_id><instance_id>6059</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>6056</service_id><instance_id>6062</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>split-brain</name><global_id>f4adcb09-1494-4073-a119-240379e5f7c0</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:1}}</description><daemon_service><service_id>6038</service_id><instance_id>6045</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:36:21</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>down+unknown</state><description>status not found</description><last_update></last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:36:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:36:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update'
2024-05-08T01:36:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:36:21'
2024-05-08T01:36:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state'
2024-05-08T01:36:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:36:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:36:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:36:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:36:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ promote_image cluster1 mirror split-brain --force
2024-05-08T01:36:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local force=--force
2024-05-08T01:36:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/split-brain --force
2024-05-08T01:36:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image promoted to primary
2024-05-08T01:36:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_stopped cluster1 mirror split-brain
2024-05-08T01:36:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_replay_state cluster1 mirror split-brain stopped
2024-05-08T01:36:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state=stopped
2024-05-08T01:36:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local s
2024-05-08T01:36:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:36:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_image_replay_state cluster1 mirror split-brain stopped
2024-05-08T01:36:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local test_state=stopped
2024-05-08T01:36:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status_result
2024-05-08T01:36:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local current_state=stopped
2024-05-08T01:36:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemons cluster1 rbd mirror status mirror/split-brain
2024-05-08T01:36:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -i state
2024-05-08T01:36:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_instance=cluster1
2024-05-08T01:36:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:36:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance=cluster1
2024-05-08T01:36:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local loop_instance
2024-05-08T01:36:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sleep 0
2024-05-08T01:36:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' cluster1 '!=' cluster1 ']'
2024-05-08T01:36:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ seq 0 3
2024-05-08T01:36:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/split-brain
2024-05-08T01:36:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:36:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ shift
2024-05-08T01:36:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:36:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:36:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:36:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:36:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:36:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:36:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:36:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local local_cluster=cluster1:0
2024-05-08T01:36:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1
2024-05-08T01:36:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:36:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:36:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:36:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=local_cluster
2024-05-08T01:36:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:36:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:36:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval local_cluster=cluster1
2024-05-08T01:36:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ local_cluster=cluster1
2024-05-08T01:36:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:36:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:36:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:36:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/split-brain
2024-05-08T01:36:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ return 0
2024-05-08T01:36:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status_result='    "state": "Stopped"'
2024-05-08T01:36:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo '    "state": "Stopped"'
2024-05-08T01:36:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -i Replaying
2024-05-08T01:36:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test stopped = stopped
2024-05-08T01:36:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+stopped
2024-05-08T01:36:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:36:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:36:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:36:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:36:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+stopped '' ''
2024-05-08T01:36:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:36:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=
2024-05-08T01:36:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:36:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:36:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:36:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain
2024-05-08T01:36:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='split-brain:
2024-05-08T01:36:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   f4adcb09-1494-4073-a119-240379e5f7c0
2024-05-08T01:36:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+stopped
2024-05-08T01:36:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: stopped
2024-05-08T01:36:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:36:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:36:26
2024-05-08T01:36:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:36:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:36:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: down+unknown
2024-05-08T01:36:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: status not found
2024-05-08T01:36:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: '
2024-05-08T01:36:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+stopped'
2024-05-08T01:36:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+stopped
2024-05-08T01:36:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*'
2024-05-08T01:36:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: stopped
2024-05-08T01:36:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:36:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:36:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:36:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:36:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:36:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:36:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:36:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:36:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:36:26'
2024-05-08T01:36:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+stopped '2024-05-08 01:36:26'
2024-05-08T01:36:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+stopped
2024-05-08T01:36:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:36:26'
2024-05-08T01:36:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:36:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:36:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:36:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>6038</service_id><instance_id>6045</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>6042</service_id><instance_id>6055</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>6049</service_id><instance_id>6059</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>6056</service_id><instance_id>6062</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>split-brain</name><global_id>f4adcb09-1494-4073-a119-240379e5f7c0</global_id><state>up+stopped</state><description>stopped</description><daemon_service><service_id>6038</service_id><instance_id>6045</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:36:26</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>down+unknown</state><description>status not found</description><last_update></last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:36:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:36:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update'
2024-05-08T01:36:28.351 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:36:26'
2024-05-08T01:36:28.352 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state'
2024-05-08T01:36:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+stopped
2024-05-08T01:36:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+stopped
2024-05-08T01:36:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+stopped
2024-05-08T01:36:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+stopped
2024-05-08T01:36:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ write_image cluster1 mirror split-brain 10
2024-05-08T01:36:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local count=10
2024-05-08T01:36:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local size=
2024-05-08T01:36:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:36:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ size=4096
2024-05-08T01:36:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster cluster1 bench mirror/split-brain --io-type write --io-size 4096 --io-threads 1 --io-total 40960 --io-pattern rand
2024-05-08T01:36:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:bench  type write io_size 4096 io_threads 1 bytes 40960 pattern random
2024-05-08T01:36:28.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2024-05-08T01:36:28.473 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:elapsed: 0   ops: 10   ops/sec: 500.011   bytes/sec: 2.0 MiB/s
2024-05-08T01:36:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ demote_image cluster1 mirror split-brain
2024-05-08T01:36:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/split-brain
2024-05-08T01:36:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Image demoted to non-primary
2024-05-08T01:36:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+error split-brain
2024-05-08T01:36:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+error
2024-05-08T01:36:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=split-brain
2024-05-08T01:36:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:36:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:36:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+error split-brain ''
2024-05-08T01:36:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+error
2024-05-08T01:36:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=split-brain
2024-05-08T01:36:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:36:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:36:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:36:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain
2024-05-08T01:36:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='split-brain:
2024-05-08T01:36:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   f4adcb09-1494-4073-a119-240379e5f7c0
2024-05-08T01:36:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+error
2024-05-08T01:36:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: split-brain detected
2024-05-08T01:36:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:36:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:36:29
2024-05-08T01:36:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:36:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:36:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: down+unknown
2024-05-08T01:36:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: status not found
2024-05-08T01:36:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: '
2024-05-08T01:36:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+error'
2024-05-08T01:36:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+error
2024-05-08T01:36:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*split-brain'
2024-05-08T01:36:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: split-brain detected
2024-05-08T01:36:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:36:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+error
2024-05-08T01:36:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:36:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+error
2024-05-08T01:36:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:36:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:36:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:36:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:36:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:36:29'
2024-05-08T01:36:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+error '2024-05-08 01:36:29'
2024-05-08T01:36:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+error
2024-05-08T01:36:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:36:29'
2024-05-08T01:36:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:36:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:36:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:36:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>6038</service_id><instance_id>6045</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>6042</service_id><instance_id>6055</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>6049</service_id><instance_id>6059</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>6056</service_id><instance_id>6062</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>split-brain</name><global_id>f4adcb09-1494-4073-a119-240379e5f7c0</global_id><state>up+error</state><description>split-brain detected</description><daemon_service><service_id>6038</service_id><instance_id>6045</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:36:29</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>down+unknown</state><description>status not found</description><last_update></last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:36:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:36:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update'
2024-05-08T01:36:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:36:29'
2024-05-08T01:36:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state'
2024-05-08T01:36:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+error
2024-05-08T01:36:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+error
2024-05-08T01:36:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+error
2024-05-08T01:36:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+error
2024-05-08T01:36:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ request_resync_image cluster1 mirror split-brain image_id
2024-05-08T01:36:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image_id_var_name=image_id
2024-05-08T01:36:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ get_image_id cluster1 mirror split-brain
2024-05-08T01:36:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local image=split-brain
2024-05-08T01:36:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster=cluster1 info mirror/split-brain
2024-05-08T01:36:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p'
2024-05-08T01:36:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval 'image_id='\''179d10f0f661'\'''
2024-05-08T01:36:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ image_id=179d10f0f661
2024-05-08T01:36:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval 'test -n "$image_id"'
2024-05-08T01:36:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ test -n 179d10f0f661
2024-05-08T01:36:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/split-brain
2024-05-08T01:36:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:Flagged image for resync from primary
2024-05-08T01:36:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror split-brain
2024-05-08T01:36:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' journal = journal ']'
2024-05-08T01:36:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position
2024-05-08T01:36:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:36:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:36:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:36:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:36:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position ''
2024-05-08T01:36:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:36:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:36:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:36:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:36:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:36:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain
2024-05-08T01:36:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='split-brain:
2024-05-08T01:36:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   f4adcb09-1494-4073-a119-240379e5f7c0
2024-05-08T01:36:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+error
2024-05-08T01:36:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: split-brain detected
2024-05-08T01:36:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:36:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:36:29
2024-05-08T01:36:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:36:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:36:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: down+unknown
2024-05-08T01:36:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: status not found
2024-05-08T01:36:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: '
2024-05-08T01:36:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:36:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:36:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:36:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position ''
2024-05-08T01:36:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:36:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:36:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:36:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:36:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:36:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain
2024-05-08T01:36:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='split-brain:
2024-05-08T01:36:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   f4adcb09-1494-4073-a119-240379e5f7c0
2024-05-08T01:36:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+error
2024-05-08T01:36:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: split-brain detected
2024-05-08T01:36:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:36:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:36:29
2024-05-08T01:36:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:36:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:36:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: down+unknown
2024-05-08T01:36:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: status not found
2024-05-08T01:36:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: '
2024-05-08T01:36:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:36:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:36:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 4
2024-05-08T01:36:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position ''
2024-05-08T01:36:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:36:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:36:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:36:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:36:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:36:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain
2024-05-08T01:36:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:rbd: error opening image split-brain: (2) No such file or directory
2024-05-08T01:36:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status=
2024-05-08T01:36:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:36:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 1
2024-05-08T01:36:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16
2024-05-08T01:36:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 8
2024-05-08T01:36:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position ''
2024-05-08T01:36:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:36:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local description_pattern=primary_position
2024-05-08T01:36:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local service_pattern=
2024-05-08T01:36:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:36:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:36:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain
2024-05-08T01:36:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='split-brain:
2024-05-08T01:36:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  global_id:   f4adcb09-1494-4073-a119-240379e5f7c0
2024-05-08T01:36:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  state:       up+replaying
2024-05-08T01:36:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:36:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  service:     mirror.0 on smithi142
2024-05-08T01:36:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  last_update: 2024-05-08 01:36:44
2024-05-08T01:36:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:  peer_sites:
2024-05-08T01:36:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    name: cluster2
2024-05-08T01:36:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    state: down+unknown
2024-05-08T01:36:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    description: status not found
2024-05-08T01:36:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:    last_update: '
2024-05-08T01:36:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  state: .*up+replaying'
2024-05-08T01:36:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  state:       up+replaying
2024-05-08T01:36:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^  description: .*primary_position'
2024-05-08T01:36:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}
2024-05-08T01:36:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n '' ']'
2024-05-08T01:36:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:36:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep '^up+'
2024-05-08T01:36:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:36:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep 'service: *mirror..* on '
2024-05-08T01:36:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:  service:     mirror.0 on smithi142
2024-05-08T01:36:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update
2024-05-08T01:36:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ sed -nEe 's/^  last_update: *(.*) *$/\1/p'
2024-05-08T01:36:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:36:44'
2024-05-08T01:36:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+replaying '2024-05-08 01:36:44'
2024-05-08T01:36:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local state_pattern=up+replaying
2024-05-08T01:36:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local 'prev_last_update=2024-05-08 01:36:44'
2024-05-08T01:36:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local status
2024-05-08T01:36:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ CEPH_ARGS=
2024-05-08T01:36:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml
2024-05-08T01:36:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ status='<status><summary><health>WARNING</health><daemon_health>OK</daemon_health><image_health>WARNING</image_health><states><unknown>1</unknown></states></summary><daemons><daemon><service_id>6038</service_id><instance_id>6045</instance_id><client_id>mirror.0</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>true</leader><health>OK</health></daemon><daemon><service_id>6042</service_id><instance_id>6055</instance_id><client_id>mirror.1</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>6049</service_id><instance_id>6059</instance_id><client_id>mirror.2</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon><daemon><service_id>6056</service_id><instance_id>6062</instance_id><client_id>mirror.3</client_id><hostname>smithi142</hostname><ceph_version>19.0.0-2529-ge0c1691a</ceph_version><leader>false</leader><health>OK</health></daemon></daemons><images><image><name>split-brain</name><global_id>f4adcb09-1494-4073-a119-240379e5f7c0</global_id><state>up+replaying</state><description>replaying, {&quot;bytes_per_second&quot;:0.0,&quot;entries_behind_primary&quot;:3,&quot;entries_per_second&quot;:0.0,&quot;non_primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:1},&quot;primary_position&quot;:{&quot;entry_tid&quot;:3,&quot;object_number&quot;:3,&quot;tag_tid&quot;:2}}</description><daemon_service><service_id>6038</service_id><instance_id>6045</instance_id><daemon_id>mirror.0</daemon_id><hostname>smithi142</hostname></daemon_service><last_update>2024-05-08 01:36:44</last_update><peer_sites><peer_site><site_name>cluster2</site_name><mirror_uuids>60884b4e-dce4-4abb-b49d-86c95c49f777</mirror_uuids><state>down+unknown</state><description>status not found</description><last_update></last_update></peer_site></peer_sites></image></images></status>'
2024-05-08T01:36:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local last_update state
2024-05-08T01:36:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update'
2024-05-08T01:36:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ last_update='2024-05-08 01:36:44'
2024-05-08T01:36:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state'
2024-05-08T01:36:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ state=up+replaying
2024-05-08T01:36:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo up+replaying
2024-05-08T01:36:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep up+replaying
2024-05-08T01:36:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:up+replaying
2024-05-08T01:36:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image_retry cluster2 mirror split-brain
2024-05-08T01:36:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 4 8 16 32
2024-05-08T01:36:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:36:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ remove_image cluster2 mirror split-brain
2024-05-08T01:36:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local image=split-brain
2024-05-08T01:36:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 snap purge mirror/split-brain
2024-05-08T01:36:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd --cluster=cluster2 rm mirror/split-brain
2024-05-08T01:36:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:
Removing image: 3% complete...
Removing image: 6% complete...
Removing image: 9% complete...
Removing image: 12% complete...
Removing image: 15% complete...
Removing image: 18% complete...
Removing image: 21% complete...
Removing image: 25% complete...
Removing image: 28% complete...
Removing image: 31% complete...
Removing image: 34% complete...
Removing image: 37% complete...
Removing image: 40% complete...
Removing image: 43% complete...
Removing image: 46% complete...
Removing image: 50% complete...
Removing image: 53% complete...
Removing image: 56% complete...
Removing image: 59% complete...
Removing image: 62% complete...
Removing image: 65% complete...
Removing image: 68% complete...
Removing image: 71% complete...
Removing image: 75% complete...
Removing image: 78% complete...
Removing image: 81% complete...
Removing image: 84% complete...
Removing image: 87% complete...
Removing image: 90% complete...
Removing image: 93% complete...
Removing image: 96% complete...
Removing image: 100% complete...done.
2024-05-08T01:36:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed'
2024-05-08T01:36:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:36:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:36:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:36:45 TEST: check if removed 'images'\''' OMAP are removed
2024-05-08T01:36:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:36:45 TEST: check if removed images' OMAP are removed
2024-05-08T01:36:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirrors cluster2
2024-05-08T01:36:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:36:45.581 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster2:0
2024-05-08T01:36:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:36:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:36:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:36:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:0
2024-05-08T01:36:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:36:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:36:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:36:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=0
2024-05-08T01:36:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 = cluster2:0 ']'
2024-05-08T01:36:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:36:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:36:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=0
2024-05-08T01:36:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:36:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:36:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster2:1
2024-05-08T01:36:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:36:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:36:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:36:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:1
2024-05-08T01:36:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:36:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:36:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:36:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=1
2024-05-08T01:36:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 = cluster2:1 ']'
2024-05-08T01:36:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:36:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:36:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=1
2024-05-08T01:36:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:36:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:36:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster2 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:36:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster2:2
2024-05-08T01:36:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:36:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:36:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:36:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:2
2024-05-08T01:36:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:36:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:36:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:36:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=2
2024-05-08T01:36:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 2 = cluster2:2 ']'
2024-05-08T01:36:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:36:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:36:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=2
2024-05-08T01:36:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:36:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:36:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster2 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:36:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ start_mirror cluster2:3
2024-05-08T01:36:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:36:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance
2024-05-08T01:36:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:36:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local val=cluster2:3
2024-05-08T01:36:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster_var_name=cluster
2024-05-08T01:36:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local instance_var_name=instance
2024-05-08T01:36:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cluster=cluster2
2024-05-08T01:36:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ instance=3
2024-05-08T01:36:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 3 = cluster2:3 ']'
2024-05-08T01:36:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval cluster=cluster2
2024-05-08T01:36:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:36:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ eval instance=3
2024-05-08T01:36:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:36:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:36:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rbd-mirror --cluster cluster2 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:36:45.729 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_in_omap cluster1 mirror
2024-05-08T01:36:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global
2024-05-08T01:36:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local obj_name=rbd_mirroring
2024-05-08T01:36:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local filter=status_global
2024-05-08T01:36:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32
2024-05-08T01:36:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:36:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:36:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global
2024-05-08T01:36:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirroring
2024-05-08T01:36:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local filter=status_global
2024-05-08T01:36:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring
2024-05-08T01:36:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -c status_global
2024-05-08T01:36:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirroring
2024-05-08T01:36:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring
2024-05-08T01:36:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 0 = 0
2024-05-08T01:36:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=0
2024-05-08T01:36:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:36:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -eq 0 ']'
2024-05-08T01:36:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_
2024-05-08T01:36:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local obj_name=rbd_mirroring
2024-05-08T01:36:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local filter=image_
2024-05-08T01:36:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32
2024-05-08T01:36:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:36:45.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:36:45.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_
2024-05-08T01:36:45.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:45.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:45.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirroring
2024-05-08T01:36:45.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local filter=image_
2024-05-08T01:36:45.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring
2024-05-08T01:36:45.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:45.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:45.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirroring
2024-05-08T01:36:45.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring
2024-05-08T01:36:45.996 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -c image_
2024-05-08T01:36:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 0 = 0
2024-05-08T01:36:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=0
2024-05-08T01:36:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:36:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -eq 0 ']'
2024-05-08T01:36:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map
2024-05-08T01:36:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local obj_name=rbd_mirror_leader
2024-05-08T01:36:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local filter=image_map
2024-05-08T01:36:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32
2024-05-08T01:36:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:36:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:36:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map
2024-05-08T01:36:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirror_leader
2024-05-08T01:36:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local filter=image_map
2024-05-08T01:36:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader
2024-05-08T01:36:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirror_leader
2024-05-08T01:36:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -c image_map
2024-05-08T01:36:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader
2024-05-08T01:36:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 1 = 0
2024-05-08T01:36:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=1
2024-05-08T01:36:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:36:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 -eq 0 ']'
2024-05-08T01:36:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32
2024-05-08T01:36:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:36:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:36:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map
2024-05-08T01:36:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirror_leader
2024-05-08T01:36:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local filter=image_map
2024-05-08T01:36:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader
2024-05-08T01:36:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirror_leader
2024-05-08T01:36:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader
2024-05-08T01:36:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -c image_map
2024-05-08T01:36:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 1 = 0
2024-05-08T01:36:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=1
2024-05-08T01:36:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:36:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 1 -eq 0 ']'
2024-05-08T01:36:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32
2024-05-08T01:36:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:36:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:36:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map
2024-05-08T01:36:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirror_leader
2024-05-08T01:36:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local filter=image_map
2024-05-08T01:36:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader
2024-05-08T01:36:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -c image_map
2024-05-08T01:36:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirror_leader
2024-05-08T01:36:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader
2024-05-08T01:36:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 0 = 0
2024-05-08T01:36:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=0
2024-05-08T01:36:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:36:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -eq 0 ']'
2024-05-08T01:36:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_image_in_omap cluster2 mirror
2024-05-08T01:36:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global
2024-05-08T01:36:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local obj_name=rbd_mirroring
2024-05-08T01:36:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local filter=status_global
2024-05-08T01:36:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32
2024-05-08T01:36:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:36:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:36:49.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global
2024-05-08T01:36:49.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:49.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:49.184 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirroring
2024-05-08T01:36:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local filter=status_global
2024-05-08T01:36:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring
2024-05-08T01:36:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -c status_global
2024-05-08T01:36:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirroring
2024-05-08T01:36:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring
2024-05-08T01:36:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 0 = 0
2024-05-08T01:36:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=0
2024-05-08T01:36:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:36:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -eq 0 ']'
2024-05-08T01:36:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_
2024-05-08T01:36:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local obj_name=rbd_mirroring
2024-05-08T01:36:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local filter=image_
2024-05-08T01:36:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32
2024-05-08T01:36:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:36:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:36:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_
2024-05-08T01:36:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirroring
2024-05-08T01:36:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local filter=image_
2024-05-08T01:36:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring
2024-05-08T01:36:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirroring
2024-05-08T01:36:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring
2024-05-08T01:36:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -c image_
2024-05-08T01:36:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 0 = 0
2024-05-08T01:36:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=0
2024-05-08T01:36:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:36:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -eq 0 ']'
2024-05-08T01:36:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map
2024-05-08T01:36:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pool=mirror
2024-05-08T01:36:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local obj_name=rbd_mirror_leader
2024-05-08T01:36:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local filter=image_map
2024-05-08T01:36:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32
2024-05-08T01:36:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 0
2024-05-08T01:36:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:36:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map
2024-05-08T01:36:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirror_leader
2024-05-08T01:36:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local filter=image_map
2024-05-08T01:36:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader
2024-05-08T01:36:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local pool=mirror
2024-05-08T01:36:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ grep -c image_map
2024-05-08T01:36:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local obj_name=rbd_mirror_leader
2024-05-08T01:36:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader
2024-05-08T01:36:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test 0 = 0
2024-05-08T01:36:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ error_code=0
2024-05-08T01:36:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set -e
2024-05-08T01:36:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -eq 0 ']'
2024-05-08T01:36:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ return 0
2024-05-08T01:36:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -z '' ']'
2024-05-08T01:36:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ testlog 'TEST: no blocklists'
2024-05-08T01:36:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ tee -a /tmp/tmp.3xddryFtVF/rbd-mirror.test.log
2024-05-08T01:36:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ date '+%F %T'
2024-05-08T01:36:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo 2024-05-08 01:36:49 TEST: no blocklists
2024-05-08T01:36:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:2024-05-08 01:36:49 TEST: no blocklists
2024-05-08T01:36:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS='--id admin'
2024-05-08T01:36:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --cluster cluster1 osd blocklist ls
2024-05-08T01:36:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -q 'listed 0 entries'
2024-05-08T01:36:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ grep -q 'listed 0 entries'
2024-05-08T01:36:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS='--id admin'
2024-05-08T01:36:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --cluster cluster2 osd blocklist ls
2024-05-08T01:36:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ cleanup 0
2024-05-08T01:36:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local error_code=0
2024-05-08T01:36:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ set +e
2024-05-08T01:36:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -ne 0 ']'
2024-05-08T01:36:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -z '' ']'
2024-05-08T01:36:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster instance
2024-05-08T01:36:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:36:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it
2024-05-08T01:36:51.466 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:pool 'mirror' does not exist
2024-05-08T01:36:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:36:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it
2024-05-08T01:36:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:pool 'mirror' does not exist
2024-05-08T01:36:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:36:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it
2024-05-08T01:36:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:pool 'mirror_parent' does not exist
2024-05-08T01:36:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ CEPH_ARGS=
2024-05-08T01:36:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it
2024-05-08T01:36:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:pool 'mirror_parent' does not exist
2024-05-08T01:36:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}"
2024-05-08T01:36:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirrors cluster1
2024-05-08T01:36:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1
2024-05-08T01:36:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:36:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:36:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster1:0 ''
2024-05-08T01:36:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:0
2024-05-08T01:36:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:36:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:36:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:36:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster1:0
2024-05-08T01:36:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1:0
2024-05-08T01:36:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:36:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:36:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:0
2024-05-08T01:36:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:36:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:36:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:36:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster1
2024-05-08T01:36:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster1
2024-05-08T01:36:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:36:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:36:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file'
2024-05-08T01:36:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid
2024-05-08T01:36:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid
2024-05-08T01:36:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=69652
2024-05-08T01:36:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 69652 ']'
2024-05-08T01:36:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 69652
2024-05-08T01:36:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:36:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:36:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:36:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=69652 '$2 == pid {print; exit 1}'
2024-05-08T01:36:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:ubuntu     69652  2.0  0.1 749532 54032 ?        Ssl  01:35   0:02 rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true
2024-05-08T01:36:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:36:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 2
2024-05-08T01:36:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:36:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=69652 '$2 == pid {print; exit 1}'
2024-05-08T01:36:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:36:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:36:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=69652 '$2 == pid {print; exit 1}'
2024-05-08T01:36:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster1
2024-05-08T01:36:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:36:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:36:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:36:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:36:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:36:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:36:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:36:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:36:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:0 cluster2
2024-05-08T01:36:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:0
2024-05-08T01:36:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 local_cluster instance
2024-05-08T01:36:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:36:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:36:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:36:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:36:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:36:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket'
2024-05-08T01:36:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok
2024-05-08T01:36:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster1:0
2024-05-08T01:36:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:0
2024-05-08T01:36:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:0 cluster instance
2024-05-08T01:36:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:0
2024-05-08T01:36:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:36:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:36:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster1:0 ']'
2024-05-08T01:36:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:36:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:36:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:36:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file'
2024-05-08T01:36:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid
2024-05-08T01:36:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid
2024-05-08T01:36:56.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:56.121 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster1:1 ''
2024-05-08T01:36:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:1
2024-05-08T01:36:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:36:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:36:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:36:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster1:1
2024-05-08T01:36:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1:1
2024-05-08T01:36:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:36:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:36:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:1
2024-05-08T01:36:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:36:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:36:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:36:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:36:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster1:1 ']'
2024-05-08T01:36:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster1
2024-05-08T01:36:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster1
2024-05-08T01:36:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:36:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:36:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file'
2024-05-08T01:36:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid
2024-05-08T01:36:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid
2024-05-08T01:36:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=69680
2024-05-08T01:36:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 69680 ']'
2024-05-08T01:36:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 69680
2024-05-08T01:36:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:36:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:36:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:36:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=69680 '$2 == pid {print; exit 1}'
2024-05-08T01:36:57.164 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:36:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:36:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=69680 '$2 == pid {print; exit 1}'
2024-05-08T01:36:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster1
2024-05-08T01:36:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:36:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:36:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:36:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:36:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:36:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:36:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:36:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:36:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:36:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:36:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:36:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:36:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:36:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:1 cluster2
2024-05-08T01:36:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:1
2024-05-08T01:36:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 local_cluster instance
2024-05-08T01:36:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:36:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:36:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:36:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:36:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:36:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:36:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:36:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:36:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket'
2024-05-08T01:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok
2024-05-08T01:36:57.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster1:1
2024-05-08T01:36:57.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:1
2024-05-08T01:36:57.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:57.234 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:1 cluster instance
2024-05-08T01:36:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:1
2024-05-08T01:36:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:36:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:36:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster1:1 ']'
2024-05-08T01:36:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:36:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:36:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:36:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:36:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file'
2024-05-08T01:36:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid
2024-05-08T01:36:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid
2024-05-08T01:36:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster1:2 ''
2024-05-08T01:36:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:2
2024-05-08T01:36:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:36:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:36:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:36:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster1:2
2024-05-08T01:36:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1:2
2024-05-08T01:36:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:36:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:36:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:2
2024-05-08T01:36:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:36:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:36:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:36:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:36:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster1:2 ']'
2024-05-08T01:36:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster1
2024-05-08T01:36:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster1
2024-05-08T01:36:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:36:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:36:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file'
2024-05-08T01:36:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid
2024-05-08T01:36:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid
2024-05-08T01:36:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=69724
2024-05-08T01:36:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 69724 ']'
2024-05-08T01:36:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 69724
2024-05-08T01:36:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:36:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:36:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:36:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=69724 '$2 == pid {print; exit 1}'
2024-05-08T01:36:58.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:36:58.300 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:36:58.301 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=69724 '$2 == pid {print; exit 1}'
2024-05-08T01:36:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster1
2024-05-08T01:36:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:36:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:36:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:36:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:36:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:36:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:36:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:36:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:36:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:36:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:36:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:36:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:36:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:36:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:2 cluster2
2024-05-08T01:36:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:2
2024-05-08T01:36:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 local_cluster instance
2024-05-08T01:36:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:36:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:36:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:36:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:36:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:36:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:36:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:36:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:36:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket'
2024-05-08T01:36:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:36:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok
2024-05-08T01:36:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster1:2
2024-05-08T01:36:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:2
2024-05-08T01:36:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:2 cluster instance
2024-05-08T01:36:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:2
2024-05-08T01:36:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:36:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:36:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster1:2 ']'
2024-05-08T01:36:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:36:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:36:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:36:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:36:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file'
2024-05-08T01:36:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid
2024-05-08T01:36:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid
2024-05-08T01:36:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster1:3 ''
2024-05-08T01:36:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster1:3
2024-05-08T01:36:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:36:58.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:36:58.393 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:36:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster1:3
2024-05-08T01:36:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster1:3
2024-05-08T01:36:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:36:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:36:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster1:3
2024-05-08T01:36:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:36:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:36:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:36:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:36:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster1:3 ']'
2024-05-08T01:36:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster1
2024-05-08T01:36:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster1
2024-05-08T01:36:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:36:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:36:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file'
2024-05-08T01:36:58.416 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid
2024-05-08T01:36:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid
2024-05-08T01:36:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=69775
2024-05-08T01:36:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 69775 ']'
2024-05-08T01:36:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 69775
2024-05-08T01:36:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:36:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:36:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:36:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=69775 '$2 == pid {print; exit 1}'
2024-05-08T01:36:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:36:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:36:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=69775 '$2 == pid {print; exit 1}'
2024-05-08T01:36:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster1
2024-05-08T01:36:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:36:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:36:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:36:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:36:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:36:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:36:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:36:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:36:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:36:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:36:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:36:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:36:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:36:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:36:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster1:3 cluster2
2024-05-08T01:36:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster1:3
2024-05-08T01:36:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:36:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 local_cluster instance
2024-05-08T01:36:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:36:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:36:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:36:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:36:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster1
2024-05-08T01:36:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster1
2024-05-08T01:36:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:36:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:36:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket'
2024-05-08T01:36:59.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:36:59.501 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok
2024-05-08T01:36:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster1:3
2024-05-08T01:36:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1:3
2024-05-08T01:36:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:36:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster1:3 cluster instance
2024-05-08T01:36:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster1:3
2024-05-08T01:36:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:36:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:36:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster1
2024-05-08T01:36:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:36:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster1:3 ']'
2024-05-08T01:36:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster1
2024-05-08T01:36:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster1
2024-05-08T01:36:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:36:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:36:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file'
2024-05-08T01:36:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid
2024-05-08T01:36:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid
2024-05-08T01:36:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}"
2024-05-08T01:36:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirrors cluster2
2024-05-08T01:36:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2
2024-05-08T01:36:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:36:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ seq 0 3
2024-05-08T01:36:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:36:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster2:0 ''
2024-05-08T01:36:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:0
2024-05-08T01:36:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:36:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:36:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:36:59.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster2:0
2024-05-08T01:36:59.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2:0
2024-05-08T01:36:59.532 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:36:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:36:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:0
2024-05-08T01:36:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:36:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:36:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:36:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:36:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 0 = cluster2:0 ']'
2024-05-08T01:36:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster2
2024-05-08T01:36:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster2
2024-05-08T01:36:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=0
2024-05-08T01:36:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=0
2024-05-08T01:36:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file'
2024-05-08T01:36:59.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid
2024-05-08T01:36:59.556 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid
2024-05-08T01:36:59.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=73579
2024-05-08T01:36:59.557 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 73579 ']'
2024-05-08T01:36:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 73579
2024-05-08T01:36:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:36:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:37:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:37:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=73579 '$2 == pid {print; exit 1}'
2024-05-08T01:37:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:37:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:37:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=73579 '$2 == pid {print; exit 1}'
2024-05-08T01:37:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster1
2024-05-08T01:37:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:37:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:37:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:37:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:37:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:37:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:37:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:37:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:37:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:37:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:37:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:37:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:37:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:37:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:37:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:37:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:37:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:37:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:0 cluster2
2024-05-08T01:37:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:0
2024-05-08T01:37:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:37:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:37:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 local_cluster instance
2024-05-08T01:37:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:37:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:37:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:37:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:37:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:37:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:37:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:37:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:37:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:37:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:37:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket'
2024-05-08T01:37:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:37:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok
2024-05-08T01:37:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster2:0
2024-05-08T01:37:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:0
2024-05-08T01:37:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:37:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:0 cluster instance
2024-05-08T01:37:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:0
2024-05-08T01:37:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:37:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:37:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:37:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=0
2024-05-08T01:37:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 0 = cluster2:0 ']'
2024-05-08T01:37:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:37:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:37:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=0
2024-05-08T01:37:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=0
2024-05-08T01:37:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file'
2024-05-08T01:37:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid
2024-05-08T01:37:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid
2024-05-08T01:37:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:37:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster2:1 ''
2024-05-08T01:37:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:1
2024-05-08T01:37:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:37:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:37:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:37:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster2:1
2024-05-08T01:37:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2:1
2024-05-08T01:37:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:37:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:37:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:1
2024-05-08T01:37:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:37:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:37:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:37:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:37:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 1 = cluster2:1 ']'
2024-05-08T01:37:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster2
2024-05-08T01:37:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster2
2024-05-08T01:37:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=1
2024-05-08T01:37:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=1
2024-05-08T01:37:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file'
2024-05-08T01:37:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid
2024-05-08T01:37:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid
2024-05-08T01:37:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=73607
2024-05-08T01:37:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 73607 ']'
2024-05-08T01:37:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 73607
2024-05-08T01:37:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:37:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:37:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:37:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=73607 '$2 == pid {print; exit 1}'
2024-05-08T01:37:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:37:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:37:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=73607 '$2 == pid {print; exit 1}'
2024-05-08T01:37:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster1
2024-05-08T01:37:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:37:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:37:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:37:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:37:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:37:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:37:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:37:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:37:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:37:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:37:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:37:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:37:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:37:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:37:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:37:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:37:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:37:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:1 cluster2
2024-05-08T01:37:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:1
2024-05-08T01:37:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:37:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:37:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 local_cluster instance
2024-05-08T01:37:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:37:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:37:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:37:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:37:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:37:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:37:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:37:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:37:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:37:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:37:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket'
2024-05-08T01:37:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:37:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok
2024-05-08T01:37:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster2:1
2024-05-08T01:37:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:1
2024-05-08T01:37:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:37:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:1 cluster instance
2024-05-08T01:37:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:1
2024-05-08T01:37:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:37:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:37:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:37:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=1
2024-05-08T01:37:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 1 = cluster2:1 ']'
2024-05-08T01:37:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:37:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:37:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=1
2024-05-08T01:37:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=1
2024-05-08T01:37:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file'
2024-05-08T01:37:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid
2024-05-08T01:37:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid
2024-05-08T01:37:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:37:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster2:2 ''
2024-05-08T01:37:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:2
2024-05-08T01:37:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:37:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:37:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:37:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster2:2
2024-05-08T01:37:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2:2
2024-05-08T01:37:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:37:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:37:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:2
2024-05-08T01:37:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:37:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:37:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:37:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:37:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 2 = cluster2:2 ']'
2024-05-08T01:37:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster2
2024-05-08T01:37:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster2
2024-05-08T01:37:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=2
2024-05-08T01:37:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=2
2024-05-08T01:37:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.2 'pid file'
2024-05-08T01:37:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid
2024-05-08T01:37:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid
2024-05-08T01:37:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=73651
2024-05-08T01:37:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 73651 ']'
2024-05-08T01:37:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 73651
2024-05-08T01:37:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:37:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:37:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:37:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=73651 '$2 == pid {print; exit 1}'
2024-05-08T01:37:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:37:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:37:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=73651 '$2 == pid {print; exit 1}'
2024-05-08T01:37:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster1
2024-05-08T01:37:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:37:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:37:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:37:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:37:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:37:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:37:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:37:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:37:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:37:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:37:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:37:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:37:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:37:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:37:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:37:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:37:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:37:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:2 cluster2
2024-05-08T01:37:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:2
2024-05-08T01:37:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:37:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:37:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 local_cluster instance
2024-05-08T01:37:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:37:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:37:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:37:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:37:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:37:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:37:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:37:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:37:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:37:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:37:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'admin socket'
2024-05-08T01:37:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:37:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok
2024-05-08T01:37:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster2:2
2024-05-08T01:37:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:2
2024-05-08T01:37:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:37:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:2 cluster instance
2024-05-08T01:37:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:2
2024-05-08T01:37:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:37:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:37:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:37:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=2
2024-05-08T01:37:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 2 = cluster2:2 ']'
2024-05-08T01:37:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:37:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:37:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=2
2024-05-08T01:37:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=2
2024-05-08T01:37:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'pid file'
2024-05-08T01:37:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid
2024-05-08T01:37:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid
2024-05-08T01:37:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}`
2024-05-08T01:37:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ stop_mirror cluster2:3 ''
2024-05-08T01:37:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local cluster=cluster2:3
2024-05-08T01:37:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local sig=
2024-05-08T01:37:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test -n ''
2024-05-08T01:37:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ local pid
2024-05-08T01:37:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ daemon_pid_file cluster2:3
2024-05-08T01:37:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster=cluster2:3
2024-05-08T01:37:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance
2024-05-08T01:37:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:37:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local val=cluster2:3
2024-05-08T01:37:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local cluster_var_name=cluster
2024-05-08T01:37:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local instance_var_name=instance
2024-05-08T01:37:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:37:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:37:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ '[' 3 = cluster2:3 ']'
2024-05-08T01:37:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval cluster=cluster2
2024-05-08T01:37:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ cluster=cluster2
2024-05-08T01:37:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ eval instance=3
2024-05-08T01:37:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ instance=3
2024-05-08T01:37:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.3 'pid file'
2024-05-08T01:37:02.948 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid
2024-05-08T01:37:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid
2024-05-08T01:37:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ pid=73711
2024-05-08T01:37:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -n 73711 ']'
2024-05-08T01:37:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ kill 73711
2024-05-08T01:37:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ for s in 1 2 4 8 16 32
2024-05-08T01:37:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ sleep 1
2024-05-08T01:37:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:37:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=73711 '$2 == pid {print; exit 1}'
2024-05-08T01:37:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ break
2024-05-08T01:37:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ ps auxww
2024-05-08T01:37:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ awk -v pid=73711 '$2 == pid {print; exit 1}'
2024-05-08T01:37:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster1
2024-05-08T01:37:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:37:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster1
2024-05-08T01:37:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:37:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:37:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:37:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:37:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:37:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:37:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:37:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:37:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:37:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:37:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:37:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:37:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:37:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:37:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:37:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_asok_file cluster2:3 cluster2
2024-05-08T01:37:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local local_cluster=cluster2:3
2024-05-08T01:37:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2
2024-05-08T01:37:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:37:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 local_cluster instance
2024-05-08T01:37:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:37:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=local_cluster
2024-05-08T01:37:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:37:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:37:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:37:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:37:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval local_cluster=cluster2
2024-05-08T01:37:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ local_cluster=cluster2
2024-05-08T01:37:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:37:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:37:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'admin socket'
2024-05-08T01:37:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:37:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok
2024-05-08T01:37:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ daemon_pid_file cluster2:3
2024-05-08T01:37:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster=cluster2:3
2024-05-08T01:37:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance
2024-05-08T01:37:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ set_cluster_instance cluster2:3 cluster instance
2024-05-08T01:37:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local val=cluster2:3
2024-05-08T01:37:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local cluster_var_name=cluster
2024-05-08T01:37:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ local instance_var_name=instance
2024-05-08T01:37:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ cluster=cluster2
2024-05-08T01:37:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ instance=3
2024-05-08T01:37:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ '[' 3 = cluster2:3 ']'
2024-05-08T01:37:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval cluster=cluster2
2024-05-08T01:37:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ cluster=cluster2
2024-05-08T01:37:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ eval instance=3
2024-05-08T01:37:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ instance=3
2024-05-08T01:37:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'pid file'
2024-05-08T01:37:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid
2024-05-08T01:37:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid
2024-05-08T01:37:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' -z 1 ']'
2024-05-08T01:37:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ test '' = /tmp/tmp.3xddryFtVF
2024-05-08T01:37:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ rm -Rf /tmp/tmp.3xddryFtVF
2024-05-08T01:37:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ '[' 0 -eq 0 ']'
2024-05-08T01:37:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ echo OK
2024-05-08T01:37:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stdout:OK
2024-05-08T01:37:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi142.stderr:+ exit 0
2024-05-08T01:37:04.058 INFO:teuthology.orchestra.run:Running command with timeout 3600
2024-05-08T01:37:04.058 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp
2024-05-08T01:37:04.069 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror.sh'] on cluster1.client.mirror...
2024-05-08T01:37:04.069 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror
2024-05-08T01:37:04.405 DEBUG:teuthology.parallel:result is None
2024-05-08T01:37:04.406 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror
2024-05-08T01:37:04.418 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror
2024-05-08T01:37:04.418 DEBUG:teuthology.orchestra.run.smithi142:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror
2024-05-08T01:37:04.473 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror
2024-05-08T01:37:04.473 DEBUG:teuthology.run_tasks:Unwinding manager ceph
2024-05-08T01:37:04.484 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean
2024-05-08T01:37:04.484 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json
2024-05-08T01:37:04.815 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:37:04.815 INFO:teuthology.orchestra.run.smithi142.stderr:dumped all
2024-05-08T01:37:04.833 INFO:teuthology.orchestra.run.smithi142.stdout:{"pg_ready":true,"pg_map":{"version":829,"stamp":"2024-05-08T01:37:04.413627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":160920,"kb_used_data":4904,"kb_used_omap":4,"kb_used_meta":155963,"kb_avail":282954600,"statfs":{"total":289910292480,"available":289745510400,"internally_reserved":0,"allocated":5021696,"data_stored":3071684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":159706447},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.551458"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.033797+0000","last_change":"2024-05-08T01:14:26.833531+0000","last_active":"2024-05-08T01:36:53.033797+0000","last_peered":"2024-05-08T01:36:53.033797+0000","last_clean":"2024-05-08T01:36:53.033797+0000","last_became_active":"2024-05-08T01:14:26.832964+0000","last_became_peered":"2024-05-08T01:14:26.832964+0000","last_unstale":"2024-05-08T01:36:53.033797+0000","last_undegraded":"2024-05-08T01:36:53.033797+0000","last_fullsized":"2024-05-08T01:36:53.033797+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:50:43.559501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.036279+0000","last_change":"2024-05-08T01:14:26.833601+0000","last_active":"2024-05-08T01:36:53.036279+0000","last_peered":"2024-05-08T01:36:53.036279+0000","last_clean":"2024-05-08T01:36:53.036279+0000","last_became_active":"2024-05-08T01:14:26.833073+0000","last_became_peered":"2024-05-08T01:14:26.833073+0000","last_unstale":"2024-05-08T01:36:53.036279+0000","last_undegraded":"2024-05-08T01:36:53.036279+0000","last_fullsized":"2024-05-08T01:36:53.036279+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T09:38:12.169806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.032259+0000","last_change":"2024-05-08T01:14:26.833188+0000","last_active":"2024-05-08T01:36:53.032259+0000","last_peered":"2024-05-08T01:36:53.032259+0000","last_clean":"2024-05-08T01:36:53.032259+0000","last_became_active":"2024-05-08T01:14:26.832760+0000","last_became_peered":"2024-05-08T01:14:26.832760+0000","last_unstale":"2024-05-08T01:36:53.032259+0000","last_undegraded":"2024-05-08T01:36:53.032259+0000","last_fullsized":"2024-05-08T01:36:53.032259+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T07:55:51.523851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.033555+0000","last_change":"2024-05-08T01:14:26.833124+0000","last_active":"2024-05-08T01:36:53.033555+0000","last_peered":"2024-05-08T01:36:53.033555+0000","last_clean":"2024-05-08T01:36:53.033555+0000","last_became_active":"2024-05-08T01:14:26.832756+0000","last_became_peered":"2024-05-08T01:14:26.832756+0000","last_unstale":"2024-05-08T01:36:53.033555+0000","last_undegraded":"2024-05-08T01:36:53.033555+0000","last_fullsized":"2024-05-08T01:36:53.033555+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T04:29:32.021107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":290,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.962947+0000","last_change":"2024-05-08T01:14:26.833102+0000","last_active":"2024-05-08T01:36:52.962947+0000","last_peered":"2024-05-08T01:36:52.962947+0000","last_clean":"2024-05-08T01:36:52.962947+0000","last_became_active":"2024-05-08T01:14:26.832771+0000","last_became_peered":"2024-05-08T01:14:26.832771+0000","last_unstale":"2024-05-08T01:36:52.962947+0000","last_undegraded":"2024-05-08T01:36:52.962947+0000","last_fullsized":"2024-05-08T01:36:52.962947+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:44:17.038811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":290,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.869461+0000","last_change":"2024-05-08T01:14:26.829716+0000","last_active":"2024-05-08T01:36:52.869461+0000","last_peered":"2024-05-08T01:36:52.869461+0000","last_clean":"2024-05-08T01:36:52.869461+0000","last_became_active":"2024-05-08T01:14:26.829273+0000","last_became_peered":"2024-05-08T01:14:26.829273+0000","last_unstale":"2024-05-08T01:36:52.869461+0000","last_undegraded":"2024-05-08T01:36:52.869461+0000","last_fullsized":"2024-05-08T01:36:52.869461+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:41:11.856331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":291,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.859280+0000","last_change":"2024-05-08T01:14:26.829079+0000","last_active":"2024-05-08T01:36:52.859280+0000","last_peered":"2024-05-08T01:36:52.859280+0000","last_clean":"2024-05-08T01:36:52.859280+0000","last_became_active":"2024-05-08T01:14:26.828803+0000","last_became_peered":"2024-05-08T01:14:26.828803+0000","last_unstale":"2024-05-08T01:36:52.859280+0000","last_undegraded":"2024-05-08T01:36:52.859280+0000","last_fullsized":"2024-05-08T01:36:52.859280+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:58:32.351165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.034942+0000","last_change":"2024-05-08T01:14:26.828790+0000","last_active":"2024-05-08T01:36:53.034942+0000","last_peered":"2024-05-08T01:36:53.034942+0000","last_clean":"2024-05-08T01:36:53.034942+0000","last_became_active":"2024-05-08T01:14:26.828455+0000","last_became_peered":"2024-05-08T01:14:26.828455+0000","last_unstale":"2024-05-08T01:36:53.034942+0000","last_undegraded":"2024-05-08T01:36:53.034942+0000","last_fullsized":"2024-05-08T01:36:53.034942+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:19:12.110097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":423,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.031556+0000","last_change":"2024-05-08T01:14:24.071859+0000","last_active":"2024-05-08T01:36:53.031556+0000","last_peered":"2024-05-08T01:36:53.031556+0000","last_clean":"2024-05-08T01:36:53.031556+0000","last_became_active":"2024-05-08T01:14:24.069533+0000","last_became_peered":"2024-05-08T01:14:24.069533+0000","last_unstale":"2024-05-08T01:36:53.031556+0000","last_undegraded":"2024-05-08T01:36:53.031556+0000","last_fullsized":"2024-05-08T01:36:53.031556+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:22.792920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:22.792920+0000","last_clean_scrub_stamp":"2024-05-08T01:14:22.792920+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:27:03.487086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738645,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58800,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313040,"statfs":{"total":96636764160,"available":96576552960,"internally_reserved":0,"allocated":1277952,"data_stored":630316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738645,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55220,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316620,"statfs":{"total":96636764160,"available":96580218880,"internally_reserved":0,"allocated":1871872,"data_stored":1220684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738645,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46900,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324940,"statfs":{"total":96636764160,"available":96588738560,"internally_reserved":0,"allocated":1871872,"data_stored":1220684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}}
2024-05-08T01:37:04.834 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json
2024-05-08T01:37:05.125 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:37:05.125 INFO:teuthology.orchestra.run.smithi142.stderr:dumped all
2024-05-08T01:37:05.144 INFO:teuthology.orchestra.run.smithi142.stdout:{"pg_ready":true,"pg_map":{"version":829,"stamp":"2024-05-08T01:37:04.413627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":160920,"kb_used_data":4904,"kb_used_omap":4,"kb_used_meta":155963,"kb_avail":282954600,"statfs":{"total":289910292480,"available":289745510400,"internally_reserved":0,"allocated":5021696,"data_stored":3071684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":159706447},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.551458"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.033797+0000","last_change":"2024-05-08T01:14:26.833531+0000","last_active":"2024-05-08T01:36:53.033797+0000","last_peered":"2024-05-08T01:36:53.033797+0000","last_clean":"2024-05-08T01:36:53.033797+0000","last_became_active":"2024-05-08T01:14:26.832964+0000","last_became_peered":"2024-05-08T01:14:26.832964+0000","last_unstale":"2024-05-08T01:36:53.033797+0000","last_undegraded":"2024-05-08T01:36:53.033797+0000","last_fullsized":"2024-05-08T01:36:53.033797+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:50:43.559501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.036279+0000","last_change":"2024-05-08T01:14:26.833601+0000","last_active":"2024-05-08T01:36:53.036279+0000","last_peered":"2024-05-08T01:36:53.036279+0000","last_clean":"2024-05-08T01:36:53.036279+0000","last_became_active":"2024-05-08T01:14:26.833073+0000","last_became_peered":"2024-05-08T01:14:26.833073+0000","last_unstale":"2024-05-08T01:36:53.036279+0000","last_undegraded":"2024-05-08T01:36:53.036279+0000","last_fullsized":"2024-05-08T01:36:53.036279+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T09:38:12.169806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.032259+0000","last_change":"2024-05-08T01:14:26.833188+0000","last_active":"2024-05-08T01:36:53.032259+0000","last_peered":"2024-05-08T01:36:53.032259+0000","last_clean":"2024-05-08T01:36:53.032259+0000","last_became_active":"2024-05-08T01:14:26.832760+0000","last_became_peered":"2024-05-08T01:14:26.832760+0000","last_unstale":"2024-05-08T01:36:53.032259+0000","last_undegraded":"2024-05-08T01:36:53.032259+0000","last_fullsized":"2024-05-08T01:36:53.032259+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T07:55:51.523851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.033555+0000","last_change":"2024-05-08T01:14:26.833124+0000","last_active":"2024-05-08T01:36:53.033555+0000","last_peered":"2024-05-08T01:36:53.033555+0000","last_clean":"2024-05-08T01:36:53.033555+0000","last_became_active":"2024-05-08T01:14:26.832756+0000","last_became_peered":"2024-05-08T01:14:26.832756+0000","last_unstale":"2024-05-08T01:36:53.033555+0000","last_undegraded":"2024-05-08T01:36:53.033555+0000","last_fullsized":"2024-05-08T01:36:53.033555+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T04:29:32.021107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":290,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.962947+0000","last_change":"2024-05-08T01:14:26.833102+0000","last_active":"2024-05-08T01:36:52.962947+0000","last_peered":"2024-05-08T01:36:52.962947+0000","last_clean":"2024-05-08T01:36:52.962947+0000","last_became_active":"2024-05-08T01:14:26.832771+0000","last_became_peered":"2024-05-08T01:14:26.832771+0000","last_unstale":"2024-05-08T01:36:52.962947+0000","last_undegraded":"2024-05-08T01:36:52.962947+0000","last_fullsized":"2024-05-08T01:36:52.962947+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:44:17.038811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":290,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.869461+0000","last_change":"2024-05-08T01:14:26.829716+0000","last_active":"2024-05-08T01:36:52.869461+0000","last_peered":"2024-05-08T01:36:52.869461+0000","last_clean":"2024-05-08T01:36:52.869461+0000","last_became_active":"2024-05-08T01:14:26.829273+0000","last_became_peered":"2024-05-08T01:14:26.829273+0000","last_unstale":"2024-05-08T01:36:52.869461+0000","last_undegraded":"2024-05-08T01:36:52.869461+0000","last_fullsized":"2024-05-08T01:36:52.869461+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:41:11.856331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":291,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.859280+0000","last_change":"2024-05-08T01:14:26.829079+0000","last_active":"2024-05-08T01:36:52.859280+0000","last_peered":"2024-05-08T01:36:52.859280+0000","last_clean":"2024-05-08T01:36:52.859280+0000","last_became_active":"2024-05-08T01:14:26.828803+0000","last_became_peered":"2024-05-08T01:14:26.828803+0000","last_unstale":"2024-05-08T01:36:52.859280+0000","last_undegraded":"2024-05-08T01:36:52.859280+0000","last_fullsized":"2024-05-08T01:36:52.859280+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:58:32.351165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.034942+0000","last_change":"2024-05-08T01:14:26.828790+0000","last_active":"2024-05-08T01:36:53.034942+0000","last_peered":"2024-05-08T01:36:53.034942+0000","last_clean":"2024-05-08T01:36:53.034942+0000","last_became_active":"2024-05-08T01:14:26.828455+0000","last_became_peered":"2024-05-08T01:14:26.828455+0000","last_unstale":"2024-05-08T01:36:53.034942+0000","last_undegraded":"2024-05-08T01:36:53.034942+0000","last_fullsized":"2024-05-08T01:36:53.034942+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:19:12.110097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":423,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.031556+0000","last_change":"2024-05-08T01:14:24.071859+0000","last_active":"2024-05-08T01:36:53.031556+0000","last_peered":"2024-05-08T01:36:53.031556+0000","last_clean":"2024-05-08T01:36:53.031556+0000","last_became_active":"2024-05-08T01:14:24.069533+0000","last_became_peered":"2024-05-08T01:14:24.069533+0000","last_unstale":"2024-05-08T01:36:53.031556+0000","last_undegraded":"2024-05-08T01:36:53.031556+0000","last_fullsized":"2024-05-08T01:36:53.031556+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:22.792920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:22.792920+0000","last_clean_scrub_stamp":"2024-05-08T01:14:22.792920+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:27:03.487086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738645,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58800,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313040,"statfs":{"total":96636764160,"available":96576552960,"internally_reserved":0,"allocated":1277952,"data_stored":630316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738645,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55220,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316620,"statfs":{"total":96636764160,"available":96580218880,"internally_reserved":0,"allocated":1871872,"data_stored":1220684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738645,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46900,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324940,"statfs":{"total":96636764160,"available":96588738560,"internally_reserved":0,"allocated":1871872,"data_stored":1220684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}}
2024-05-08T01:37:05.145 INFO:tasks.ceph.ceph_manager.cluster2:clean!
2024-05-08T01:37:05.145 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json
2024-05-08T01:37:05.438 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:37:05.441 INFO:teuthology.orchestra.run.smithi142.stderr:dumped all
2024-05-08T01:37:05.456 INFO:teuthology.orchestra.run.smithi142.stdout:{"pg_ready":true,"pg_map":{"version":829,"stamp":"2024-05-08T01:37:04.413627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":160920,"kb_used_data":4904,"kb_used_omap":4,"kb_used_meta":155963,"kb_avail":282954600,"statfs":{"total":289910292480,"available":289745510400,"internally_reserved":0,"allocated":5021696,"data_stored":3071684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":159706447},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.551458"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.033797+0000","last_change":"2024-05-08T01:14:26.833531+0000","last_active":"2024-05-08T01:36:53.033797+0000","last_peered":"2024-05-08T01:36:53.033797+0000","last_clean":"2024-05-08T01:36:53.033797+0000","last_became_active":"2024-05-08T01:14:26.832964+0000","last_became_peered":"2024-05-08T01:14:26.832964+0000","last_unstale":"2024-05-08T01:36:53.033797+0000","last_undegraded":"2024-05-08T01:36:53.033797+0000","last_fullsized":"2024-05-08T01:36:53.033797+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:50:43.559501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.036279+0000","last_change":"2024-05-08T01:14:26.833601+0000","last_active":"2024-05-08T01:36:53.036279+0000","last_peered":"2024-05-08T01:36:53.036279+0000","last_clean":"2024-05-08T01:36:53.036279+0000","last_became_active":"2024-05-08T01:14:26.833073+0000","last_became_peered":"2024-05-08T01:14:26.833073+0000","last_unstale":"2024-05-08T01:36:53.036279+0000","last_undegraded":"2024-05-08T01:36:53.036279+0000","last_fullsized":"2024-05-08T01:36:53.036279+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T09:38:12.169806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.032259+0000","last_change":"2024-05-08T01:14:26.833188+0000","last_active":"2024-05-08T01:36:53.032259+0000","last_peered":"2024-05-08T01:36:53.032259+0000","last_clean":"2024-05-08T01:36:53.032259+0000","last_became_active":"2024-05-08T01:14:26.832760+0000","last_became_peered":"2024-05-08T01:14:26.832760+0000","last_unstale":"2024-05-08T01:36:53.032259+0000","last_undegraded":"2024-05-08T01:36:53.032259+0000","last_fullsized":"2024-05-08T01:36:53.032259+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T07:55:51.523851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.033555+0000","last_change":"2024-05-08T01:14:26.833124+0000","last_active":"2024-05-08T01:36:53.033555+0000","last_peered":"2024-05-08T01:36:53.033555+0000","last_clean":"2024-05-08T01:36:53.033555+0000","last_became_active":"2024-05-08T01:14:26.832756+0000","last_became_peered":"2024-05-08T01:14:26.832756+0000","last_unstale":"2024-05-08T01:36:53.033555+0000","last_undegraded":"2024-05-08T01:36:53.033555+0000","last_fullsized":"2024-05-08T01:36:53.033555+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T04:29:32.021107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":290,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.962947+0000","last_change":"2024-05-08T01:14:26.833102+0000","last_active":"2024-05-08T01:36:52.962947+0000","last_peered":"2024-05-08T01:36:52.962947+0000","last_clean":"2024-05-08T01:36:52.962947+0000","last_became_active":"2024-05-08T01:14:26.832771+0000","last_became_peered":"2024-05-08T01:14:26.832771+0000","last_unstale":"2024-05-08T01:36:52.962947+0000","last_undegraded":"2024-05-08T01:36:52.962947+0000","last_fullsized":"2024-05-08T01:36:52.962947+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:44:17.038811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":290,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.869461+0000","last_change":"2024-05-08T01:14:26.829716+0000","last_active":"2024-05-08T01:36:52.869461+0000","last_peered":"2024-05-08T01:36:52.869461+0000","last_clean":"2024-05-08T01:36:52.869461+0000","last_became_active":"2024-05-08T01:14:26.829273+0000","last_became_peered":"2024-05-08T01:14:26.829273+0000","last_unstale":"2024-05-08T01:36:52.869461+0000","last_undegraded":"2024-05-08T01:36:52.869461+0000","last_fullsized":"2024-05-08T01:36:52.869461+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:41:11.856331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":291,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.859280+0000","last_change":"2024-05-08T01:14:26.829079+0000","last_active":"2024-05-08T01:36:52.859280+0000","last_peered":"2024-05-08T01:36:52.859280+0000","last_clean":"2024-05-08T01:36:52.859280+0000","last_became_active":"2024-05-08T01:14:26.828803+0000","last_became_peered":"2024-05-08T01:14:26.828803+0000","last_unstale":"2024-05-08T01:36:52.859280+0000","last_undegraded":"2024-05-08T01:36:52.859280+0000","last_fullsized":"2024-05-08T01:36:52.859280+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:58:32.351165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.034942+0000","last_change":"2024-05-08T01:14:26.828790+0000","last_active":"2024-05-08T01:36:53.034942+0000","last_peered":"2024-05-08T01:36:53.034942+0000","last_clean":"2024-05-08T01:36:53.034942+0000","last_became_active":"2024-05-08T01:14:26.828455+0000","last_became_peered":"2024-05-08T01:14:26.828455+0000","last_unstale":"2024-05-08T01:36:53.034942+0000","last_undegraded":"2024-05-08T01:36:53.034942+0000","last_fullsized":"2024-05-08T01:36:53.034942+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:19:12.110097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":423,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.031556+0000","last_change":"2024-05-08T01:14:24.071859+0000","last_active":"2024-05-08T01:36:53.031556+0000","last_peered":"2024-05-08T01:36:53.031556+0000","last_clean":"2024-05-08T01:36:53.031556+0000","last_became_active":"2024-05-08T01:14:24.069533+0000","last_became_peered":"2024-05-08T01:14:24.069533+0000","last_unstale":"2024-05-08T01:36:53.031556+0000","last_undegraded":"2024-05-08T01:36:53.031556+0000","last_fullsized":"2024-05-08T01:36:53.031556+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:22.792920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:22.792920+0000","last_clean_scrub_stamp":"2024-05-08T01:14:22.792920+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:27:03.487086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738645,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58800,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313040,"statfs":{"total":96636764160,"available":96576552960,"internally_reserved":0,"allocated":1277952,"data_stored":630316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738645,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55220,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316620,"statfs":{"total":96636764160,"available":96580218880,"internally_reserved":0,"allocated":1871872,"data_stored":1220684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738645,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46900,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324940,"statfs":{"total":96636764160,"available":96588738560,"internally_reserved":0,"allocated":1871872,"data_stored":1220684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}}
2024-05-08T01:37:05.457 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json
2024-05-08T01:37:05.745 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:37:05.745 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":149,"fsid":"e7480526-62e8-4ed0-8efe-58a8e36c6942","created":"2024-05-08T01:14:10.569210+0000","modified":"2024-05-08T01:36:52.854778+0000","last_up_change":"2024-05-08T01:14:20.782949+0000","last_in_change":"2024-05-08T01:14:11.492301+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T01:14:21.961770+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-08T01:14:25.806066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b037d45-97f6-4d6b-9827-0fae9be5eede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":420380403},{"type":"v1","addr":"172.21.15.142:6807","nonce":420380403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":420380403},{"type":"v1","addr":"172.21.15.142:6813","nonce":420380403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6821","nonce":420380403},{"type":"v1","addr":"172.21.15.142:6823","nonce":420380403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":420380403},{"type":"v1","addr":"172.21.15.142:6819","nonce":420380403}]},"public_addr":"172.21.15.142:6807/420380403","cluster_addr":"172.21.15.142:6813/420380403","heartbeat_back_addr":"172.21.15.142:6823/420380403","heartbeat_front_addr":"172.21.15.142:6819/420380403","state":["exists","up"]},{"osd":1,"uuid":"b14aaee3-26c1-43d8-bd53-f04a32fb66de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":3687546466},{"type":"v1","addr":"172.21.15.142:6801","nonce":3687546466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":3687546466},{"type":"v1","addr":"172.21.15.142:6805","nonce":3687546466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":3687546466},{"type":"v1","addr":"172.21.15.142:6817","nonce":3687546466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":3687546466},{"type":"v1","addr":"172.21.15.142:6811","nonce":3687546466}]},"public_addr":"172.21.15.142:6801/3687546466","cluster_addr":"172.21.15.142:6805/3687546466","heartbeat_back_addr":"172.21.15.142:6817/3687546466","heartbeat_front_addr":"172.21.15.142:6811/3687546466","state":["exists","up"]},{"osd":2,"uuid":"308ad5c2-db06-434a-8986-73794c7f77e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6803","nonce":2528577693},{"type":"v1","addr":"172.21.15.142:6806","nonce":2528577693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6809","nonce":2528577693},{"type":"v1","addr":"172.21.15.142:6812","nonce":2528577693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2528577693},{"type":"v1","addr":"172.21.15.142:6822","nonce":2528577693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6815","nonce":2528577693},{"type":"v1","addr":"172.21.15.142:6818","nonce":2528577693}]},"public_addr":"172.21.15.142:6806/2528577693","cluster_addr":"172.21.15.142:6812/2528577693","heartbeat_back_addr":"172.21.15.142:6822/2528577693","heartbeat_front_addr":"172.21.15.142:6818/2528577693","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:14:16.954207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:14:16.922438+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:14:17.044924+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
2024-05-08T01:37:06.763 INFO:tasks.ceph:Scrubbing cluster2.osd.0
2024-05-08T01:37:06.763 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0
2024-05-08T01:37:06.920 INFO:teuthology.orchestra.run.smithi142.stdout:{
2024-05-08T01:37:06.920 INFO:teuthology.orchestra.run.smithi142.stdout:    "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) "
2024-05-08T01:37:06.921 INFO:teuthology.orchestra.run.smithi142.stdout:}
2024-05-08T01:37:06.937 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0
2024-05-08T01:37:07.225 INFO:teuthology.orchestra.run.smithi142.stderr:instructed osd(s) 0 to deep-scrub
2024-05-08T01:37:07.244 INFO:tasks.ceph:Scrubbing cluster2.osd.1
2024-05-08T01:37:07.244 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0
2024-05-08T01:37:07.385 INFO:teuthology.orchestra.run.smithi142.stdout:{
2024-05-08T01:37:07.385 INFO:teuthology.orchestra.run.smithi142.stdout:    "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) "
2024-05-08T01:37:07.385 INFO:teuthology.orchestra.run.smithi142.stdout:}
2024-05-08T01:37:07.400 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1
2024-05-08T01:37:07.663 INFO:teuthology.orchestra.run.smithi142.stderr:instructed osd(s) 1 to deep-scrub
2024-05-08T01:37:07.680 INFO:tasks.ceph:Scrubbing cluster2.osd.2
2024-05-08T01:37:07.680 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0
2024-05-08T01:37:07.816 INFO:teuthology.orchestra.run.smithi142.stdout:{
2024-05-08T01:37:07.816 INFO:teuthology.orchestra.run.smithi142.stdout:    "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) "
2024-05-08T01:37:07.816 INFO:teuthology.orchestra.run.smithi142.stdout:}
2024-05-08T01:37:07.831 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2
2024-05-08T01:37:08.109 INFO:teuthology.orchestra.run.smithi142.stderr:instructed osd(s) 2 to deep-scrub
2024-05-08T01:37:08.127 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json
2024-05-08T01:37:08.414 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:37:08.414 INFO:teuthology.orchestra.run.smithi142.stderr:dumped all
2024-05-08T01:37:08.431 INFO:teuthology.orchestra.run.smithi142.stdout:{"pg_ready":true,"pg_map":{"version":831,"stamp":"2024-05-08T01:37:08.414650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":160920,"kb_used_data":4904,"kb_used_omap":4,"kb_used_meta":155963,"kb_avail":282954600,"statfs":{"total":289910292480,"available":289745510400,"internally_reserved":0,"allocated":5021696,"data_stored":3071684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":159706447},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003042"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.033797+0000","last_change":"2024-05-08T01:14:26.833531+0000","last_active":"2024-05-08T01:36:53.033797+0000","last_peered":"2024-05-08T01:36:53.033797+0000","last_clean":"2024-05-08T01:36:53.033797+0000","last_became_active":"2024-05-08T01:14:26.832964+0000","last_became_peered":"2024-05-08T01:14:26.832964+0000","last_unstale":"2024-05-08T01:36:53.033797+0000","last_undegraded":"2024-05-08T01:36:53.033797+0000","last_fullsized":"2024-05-08T01:36:53.033797+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:50:43.559501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.036279+0000","last_change":"2024-05-08T01:14:26.833601+0000","last_active":"2024-05-08T01:36:53.036279+0000","last_peered":"2024-05-08T01:36:53.036279+0000","last_clean":"2024-05-08T01:36:53.036279+0000","last_became_active":"2024-05-08T01:14:26.833073+0000","last_became_peered":"2024-05-08T01:14:26.833073+0000","last_unstale":"2024-05-08T01:36:53.036279+0000","last_undegraded":"2024-05-08T01:36:53.036279+0000","last_fullsized":"2024-05-08T01:36:53.036279+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T09:38:12.169806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.032259+0000","last_change":"2024-05-08T01:14:26.833188+0000","last_active":"2024-05-08T01:36:53.032259+0000","last_peered":"2024-05-08T01:36:53.032259+0000","last_clean":"2024-05-08T01:36:53.032259+0000","last_became_active":"2024-05-08T01:14:26.832760+0000","last_became_peered":"2024-05-08T01:14:26.832760+0000","last_unstale":"2024-05-08T01:36:53.032259+0000","last_undegraded":"2024-05-08T01:36:53.032259+0000","last_fullsized":"2024-05-08T01:36:53.032259+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T07:55:51.523851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.033555+0000","last_change":"2024-05-08T01:14:26.833124+0000","last_active":"2024-05-08T01:36:53.033555+0000","last_peered":"2024-05-08T01:36:53.033555+0000","last_clean":"2024-05-08T01:36:53.033555+0000","last_became_active":"2024-05-08T01:14:26.832756+0000","last_became_peered":"2024-05-08T01:14:26.832756+0000","last_unstale":"2024-05-08T01:36:53.033555+0000","last_undegraded":"2024-05-08T01:36:53.033555+0000","last_fullsized":"2024-05-08T01:36:53.033555+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T04:29:32.021107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":290,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.962947+0000","last_change":"2024-05-08T01:14:26.833102+0000","last_active":"2024-05-08T01:36:52.962947+0000","last_peered":"2024-05-08T01:36:52.962947+0000","last_clean":"2024-05-08T01:36:52.962947+0000","last_became_active":"2024-05-08T01:14:26.832771+0000","last_became_peered":"2024-05-08T01:14:26.832771+0000","last_unstale":"2024-05-08T01:36:52.962947+0000","last_undegraded":"2024-05-08T01:36:52.962947+0000","last_fullsized":"2024-05-08T01:36:52.962947+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:44:17.038811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":290,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.869461+0000","last_change":"2024-05-08T01:14:26.829716+0000","last_active":"2024-05-08T01:36:52.869461+0000","last_peered":"2024-05-08T01:36:52.869461+0000","last_clean":"2024-05-08T01:36:52.869461+0000","last_became_active":"2024-05-08T01:14:26.829273+0000","last_became_peered":"2024-05-08T01:14:26.829273+0000","last_unstale":"2024-05-08T01:36:52.869461+0000","last_undegraded":"2024-05-08T01:36:52.869461+0000","last_fullsized":"2024-05-08T01:36:52.869461+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:41:11.856331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":291,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.859280+0000","last_change":"2024-05-08T01:14:26.829079+0000","last_active":"2024-05-08T01:36:52.859280+0000","last_peered":"2024-05-08T01:36:52.859280+0000","last_clean":"2024-05-08T01:36:52.859280+0000","last_became_active":"2024-05-08T01:14:26.828803+0000","last_became_peered":"2024-05-08T01:14:26.828803+0000","last_unstale":"2024-05-08T01:36:52.859280+0000","last_undegraded":"2024-05-08T01:36:52.859280+0000","last_fullsized":"2024-05-08T01:36:52.859280+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:58:32.351165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":289,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.034942+0000","last_change":"2024-05-08T01:14:26.828790+0000","last_active":"2024-05-08T01:36:53.034942+0000","last_peered":"2024-05-08T01:36:53.034942+0000","last_clean":"2024-05-08T01:36:53.034942+0000","last_became_active":"2024-05-08T01:14:26.828455+0000","last_became_peered":"2024-05-08T01:14:26.828455+0000","last_unstale":"2024-05-08T01:36:53.034942+0000","last_undegraded":"2024-05-08T01:36:53.034942+0000","last_fullsized":"2024-05-08T01:36:53.034942+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:25.809843+0000","last_clean_scrub_stamp":"2024-05-08T01:14:25.809843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:19:12.110097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":423,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.031556+0000","last_change":"2024-05-08T01:14:24.071859+0000","last_active":"2024-05-08T01:36:53.031556+0000","last_peered":"2024-05-08T01:36:53.031556+0000","last_clean":"2024-05-08T01:36:53.031556+0000","last_became_active":"2024-05-08T01:14:24.069533+0000","last_became_peered":"2024-05-08T01:14:24.069533+0000","last_unstale":"2024-05-08T01:36:53.031556+0000","last_undegraded":"2024-05-08T01:36:53.031556+0000","last_fullsized":"2024-05-08T01:36:53.031556+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:14:22.792920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:14:22.792920+0000","last_clean_scrub_stamp":"2024-05-08T01:14:22.792920+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:27:03.487086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738646,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58800,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313040,"statfs":{"total":96636764160,"available":96576552960,"internally_reserved":0,"allocated":1277952,"data_stored":630316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738646,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55220,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316620,"statfs":{"total":96636764160,"available":96580218880,"internally_reserved":0,"allocated":1871872,"data_stored":1220684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738646,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46900,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324940,"statfs":{"total":96636764160,"available":96588738560,"internally_reserved":0,"allocated":1871872,"data_stored":1220684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}}
2024-05-08T01:37:08.435 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-05-08T01:14:25.809843+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=14, tm_sec=25, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=37, tm_sec=5, tm_wday=2, tm_yday=129, tm_isdst=0)
2024-05-08T01:37:08.435 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-05-08T01:14:25.809843+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=14, tm_sec=25, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=37, tm_sec=5, tm_wday=2, tm_yday=129, tm_isdst=0)
2024-05-08T01:37:08.435 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-05-08T01:14:25.809843+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=14, tm_sec=25, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=37, tm_sec=5, tm_wday=2, tm_yday=129, tm_isdst=0)
2024-05-08T01:37:08.435 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-05-08T01:14:25.809843+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=14, tm_sec=25, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=37, tm_sec=5, tm_wday=2, tm_yday=129, tm_isdst=0)
2024-05-08T01:37:08.435 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-05-08T01:14:25.809843+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=14, tm_sec=25, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=37, tm_sec=5, tm_wday=2, tm_yday=129, tm_isdst=0)
2024-05-08T01:37:08.435 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-05-08T01:14:25.809843+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=14, tm_sec=25, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=37, tm_sec=5, tm_wday=2, tm_yday=129, tm_isdst=0)
2024-05-08T01:37:08.436 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-05-08T01:14:25.809843+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=14, tm_sec=25, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=37, tm_sec=5, tm_wday=2, tm_yday=129, tm_isdst=0)
2024-05-08T01:37:08.436 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-05-08T01:14:25.809843+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=14, tm_sec=25, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=37, tm_sec=5, tm_wday=2, tm_yday=129, tm_isdst=0)
2024-05-08T01:37:08.436 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-05-08T01:14:22.792920+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=14, tm_sec=22, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=37, tm_sec=5, tm_wday=2, tm_yday=129, tm_isdst=0)
2024-05-08T01:37:08.436 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed.
2024-05-08T01:37:28.437 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json
2024-05-08T01:37:28.741 INFO:teuthology.orchestra.run.smithi142.stdout:
2024-05-08T01:37:28.741 INFO:teuthology.orchestra.run.smithi142.stderr:dumped all
2024-05-08T01:37:28.760 INFO:teuthology.orchestra.run.smithi142.stdout:{"pg_ready":true,"pg_map":{"version":841,"stamp":"2024-05-08T01:37:28.419460+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":160896,"kb_used_data":4904,"kb_used_omap":4,"kb_used_meta":155963,"kb_avail":282954624,"statfs":{"total":289910292480,"available":289745534976,"internally_reserved":0,"allocated":5021696,"data_stored":3071684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":159706447},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002968"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":298,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:37:12.461770+0000","last_change":"2024-05-08T01:37:12.461822+0000","last_active":"2024-05-08T01:37:12.461770+0000","last_peered":"2024-05-08T01:37:12.461770+0000","last_clean":"2024-05-08T01:37:12.461770+0000","last_became_active":"2024-05-08T01:14:26.832964+0000","last_became_peered":"2024-05-08T01:14:26.832964+0000","last_unstale":"2024-05-08T01:37:12.461770+0000","last_undegraded":"2024-05-08T01:37:12.461770+0000","last_fullsized":"2024-05-08T01:37:12.461770+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:37:12.461578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:37:12.461578+0000","last_clean_scrub_stamp":"2024-05-08T01:37:12.461578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T01:46:04.535498+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":298,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:37:13.497174+0000","last_change":"2024-05-08T01:37:13.497227+0000","last_active":"2024-05-08T01:37:13.497174+0000","last_peered":"2024-05-08T01:37:13.497174+0000","last_clean":"2024-05-08T01:37:13.497174+0000","last_became_active":"2024-05-08T01:14:26.833073+0000","last_became_peered":"2024-05-08T01:14:26.833073+0000","last_unstale":"2024-05-08T01:37:13.497174+0000","last_undegraded":"2024-05-08T01:37:13.497174+0000","last_fullsized":"2024-05-08T01:37:13.497174+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:37:13.496974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:37:13.496974+0000","last_clean_scrub_stamp":"2024-05-08T01:37:13.496974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T09:02:05.638447+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":299,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:37:11.460683+0000","last_change":"2024-05-08T01:37:11.460683+0000","last_active":"2024-05-08T01:37:11.460683+0000","last_peered":"2024-05-08T01:37:11.460683+0000","last_clean":"2024-05-08T01:37:11.460683+0000","last_became_active":"2024-05-08T01:14:26.832760+0000","last_became_peered":"2024-05-08T01:14:26.832760+0000","last_unstale":"2024-05-08T01:37:11.460683+0000","last_undegraded":"2024-05-08T01:37:11.460683+0000","last_fullsized":"2024-05-08T01:37:11.460683+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:37:11.460445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:37:11.460445+0000","last_clean_scrub_stamp":"2024-05-08T01:37:11.460445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:33:44.021092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":299,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:37:10.466586+0000","last_change":"2024-05-08T01:37:10.466586+0000","last_active":"2024-05-08T01:37:10.466586+0000","last_peered":"2024-05-08T01:37:10.466586+0000","last_clean":"2024-05-08T01:37:10.466586+0000","last_became_active":"2024-05-08T01:14:26.832756+0000","last_became_peered":"2024-05-08T01:14:26.832756+0000","last_unstale":"2024-05-08T01:37:10.466586+0000","last_undegraded":"2024-05-08T01:37:10.466586+0000","last_fullsized":"2024-05-08T01:37:10.466586+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:37:10.466290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:37:10.466290+0000","last_clean_scrub_stamp":"2024-05-08T01:37:10.466290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:38:33.488664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":300,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:37:08.069799+0000","last_change":"2024-05-08T01:37:08.069799+0000","last_active":"2024-05-08T01:37:08.069799+0000","last_peered":"2024-05-08T01:37:08.069799+0000","last_clean":"2024-05-08T01:37:08.069799+0000","last_became_active":"2024-05-08T01:14:26.832771+0000","last_became_peered":"2024-05-08T01:14:26.832771+0000","last_unstale":"2024-05-08T01:37:08.069799+0000","last_undegraded":"2024-05-08T01:37:08.069799+0000","last_fullsized":"2024-05-08T01:37:08.069799+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:37:08.069637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:37:08.069637+0000","last_clean_scrub_stamp":"2024-05-08T01:37:08.069637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T11:48:50.151590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":300,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:37:09.824631+0000","last_change":"2024-05-08T01:37:09.824631+0000","last_active":"2024-05-08T01:37:09.824631+0000","last_peered":"2024-05-08T01:37:09.824631+0000","last_clean":"2024-05-08T01:37:09.824631+0000","last_became_active":"2024-05-08T01:14:26.829273+0000","last_became_peered":"2024-05-08T01:14:26.829273+0000","last_unstale":"2024-05-08T01:37:09.824631+0000","last_undegraded":"2024-05-08T01:37:09.824631+0000","last_fullsized":"2024-05-08T01:37:09.824631+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:37:09.824389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:37:09.824389+0000","last_clean_scrub_stamp":"2024-05-08T01:37:09.824389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:58:22.910499+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":300,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:37:08.866094+0000","last_change":"2024-05-08T01:37:08.866153+0000","last_active":"2024-05-08T01:37:08.866094+0000","last_peered":"2024-05-08T01:37:08.866094+0000","last_clean":"2024-05-08T01:37:08.866094+0000","last_became_active":"2024-05-08T01:14:26.828803+0000","last_became_peered":"2024-05-08T01:14:26.828803+0000","last_unstale":"2024-05-08T01:37:08.866094+0000","last_undegraded":"2024-05-08T01:37:08.866094+0000","last_fullsized":"2024-05-08T01:37:08.866094+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:37:08.865906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:37:08.865906+0000","last_clean_scrub_stamp":"2024-05-08T01:37:08.865906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:35:13.945945+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":299,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:37:09.478228+0000","last_change":"2024-05-08T01:37:09.478228+0000","last_active":"2024-05-08T01:37:09.478228+0000","last_peered":"2024-05-08T01:37:09.478228+0000","last_clean":"2024-05-08T01:37:09.478228+0000","last_became_active":"2024-05-08T01:14:26.828455+0000","last_became_peered":"2024-05-08T01:14:26.828455+0000","last_unstale":"2024-05-08T01:37:09.478228+0000","last_undegraded":"2024-05-08T01:37:09.478228+0000","last_fullsized":"2024-05-08T01:37:09.478228+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:37:09.477891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:37:09.477891+0000","last_clean_scrub_stamp":"2024-05-08T01:37:09.477891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T07:15:58.453070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"149'99","reported_seq":434,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-05-08T01:37:08.432290+0000","last_change":"2024-05-08T01:37:08.432366+0000","last_active":"2024-05-08T01:37:08.432290+0000","last_peered":"2024-05-08T01:37:08.432290+0000","last_clean":"2024-05-08T01:37:08.432290+0000","last_became_active":"2024-05-08T01:14:24.069533+0000","last_became_peered":"2024-05-08T01:14:24.069533+0000","last_unstale":"2024-05-08T01:37:08.432290+0000","last_undegraded":"2024-05-08T01:37:08.432290+0000","last_fullsized":"2024-05-08T01:37:08.432290+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'99","last_scrub_stamp":"2024-05-08T01:37:08.432086+0000","last_deep_scrub":"149'99","last_deep_scrub_stamp":"2024-05-08T01:37:08.432086+0000","last_clean_scrub_stamp":"2024-05-08T01:37:08.432086+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:48:50.769774+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738650,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58792,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313048,"statfs":{"total":96636764160,"available":96576561152,"internally_reserved":0,"allocated":1277952,"data_stored":630316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738650,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55212,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316628,"statfs":{"total":96636764160,"available":96580227072,"internally_reserved":0,"allocated":1871872,"data_stored":1220684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738650,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46892,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324948,"statfs":{"total":96636764160,"available":96588746752,"internally_reserved":0,"allocated":1871872,"data_stored":1220684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}}
2024-05-08T01:37:28.761 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false
2024-05-08T01:37:29.057 INFO:teuthology.misc:Shutting down mds daemons...
2024-05-08T01:37:29.057 INFO:teuthology.misc:Shutting down osd daemons...
2024-05-08T01:37:29.057 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit
2024-05-08T01:37:29.057 INFO:teuthology.orchestra.run:waiting for 300
2024-05-08T01:37:29.126 INFO:tasks.ceph.cluster2.osd.0:Stopped
2024-05-08T01:37:29.126 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit
2024-05-08T01:37:29.126 INFO:teuthology.orchestra.run:waiting for 300
2024-05-08T01:37:29.208 INFO:tasks.ceph.cluster2.osd.1:Stopped
2024-05-08T01:37:29.208 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit
2024-05-08T01:37:29.208 INFO:teuthology.orchestra.run:waiting for 300
2024-05-08T01:37:29.296 INFO:tasks.ceph.cluster2.osd.2:Stopped
2024-05-08T01:37:29.296 INFO:teuthology.misc:Shutting down mgr daemons...
2024-05-08T01:37:29.297 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit
2024-05-08T01:37:29.297 INFO:teuthology.orchestra.run:waiting for 300
2024-05-08T01:37:29.325 INFO:tasks.ceph.cluster2.mgr.x:Stopped
2024-05-08T01:37:29.325 INFO:teuthology.misc:Shutting down mon daemons...
2024-05-08T01:37:29.326 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit
2024-05-08T01:37:29.326 INFO:teuthology.orchestra.run:waiting for 300
2024-05-08T01:37:29.343 INFO:tasks.ceph.cluster2.mon.a:Stopped
2024-05-08T01:37:29.344 INFO:tasks.ceph:Checking cluster log for badness...
2024-05-08T01:37:29.344 DEBUG:teuthology.orchestra.run.smithi142:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1
2024-05-08T01:37:29.399 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi142.front.sepia.ceph.com
2024-05-08T01:37:29.399 DEBUG:teuthology.orchestra.run.smithi142:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0
2024-05-08T01:37:29.655 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi142.front.sepia.ceph.com
2024-05-08T01:37:29.655 DEBUG:teuthology.orchestra.run.smithi142:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1
2024-05-08T01:37:29.853 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi142.front.sepia.ceph.com
2024-05-08T01:37:29.854 DEBUG:teuthology.orchestra.run.smithi142:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2
2024-05-08T01:37:30.027 INFO:tasks.ceph:Archiving mon data...
2024-05-08T01:37:30.029 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696665/data/cluster2.mon.a.tgz
2024-05-08T01:37:30.029 DEBUG:teuthology.orchestra.run.smithi142:> mktemp
2024-05-08T01:37:30.034 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.8Qw64xgDhg
2024-05-08T01:37:30.034 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.8Qw64xgDhg
2024-05-08T01:37:30.171 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.8Qw64xgDhg
2024-05-08T01:37:30.238 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.8Qw64xgDhg is 378KB
2024-05-08T01:37:30.299 DEBUG:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.8Qw64xgDhg
2024-05-08T01:37:30.304 INFO:tasks.ceph:Cleaning ceph cluster...
2024-05-08T01:37:30.304 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid
2024-05-08T01:37:30.307 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid
2024-05-08T01:37:30.362 DEBUG:teuthology.run_tasks:Unwinding manager ceph
2024-05-08T01:37:30.376 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean
2024-05-08T01:37:30.376 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json
2024-05-08T01:37:30.682 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:37:30.683 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all
2024-05-08T01:37:30.701 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":865,"stamp":"2024-05-08T01:37:29.342503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":134680,"kb_used_data":5036,"kb_used_omap":4,"kb_used_meta":129595,"kb_avail":282980840,"statfs":{"total":289910292480,"available":289772380160,"internally_reserved":0,"allocated":5156864,"data_stored":3155669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":132705615},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002810"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":302,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.602190+0000","last_change":"2024-05-08T01:13:50.943750+0000","last_active":"2024-05-08T01:36:53.602190+0000","last_peered":"2024-05-08T01:36:53.602190+0000","last_clean":"2024-05-08T01:36:53.602190+0000","last_became_active":"2024-05-08T01:13:50.943014+0000","last_became_peered":"2024-05-08T01:13:50.943014+0000","last_unstale":"2024-05-08T01:36:53.602190+0000","last_undegraded":"2024-05-08T01:36:53.602190+0000","last_fullsized":"2024-05-08T01:36:53.602190+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:26:50.611238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":302,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.606876+0000","last_change":"2024-05-08T01:13:50.945953+0000","last_active":"2024-05-08T01:36:53.606876+0000","last_peered":"2024-05-08T01:36:53.606876+0000","last_clean":"2024-05-08T01:36:53.606876+0000","last_became_active":"2024-05-08T01:13:50.945427+0000","last_became_peered":"2024-05-08T01:13:50.945427+0000","last_unstale":"2024-05-08T01:36:53.606876+0000","last_undegraded":"2024-05-08T01:36:53.606876+0000","last_fullsized":"2024-05-08T01:36:53.606876+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:26:08.751583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":303,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.601275+0000","last_change":"2024-05-08T01:13:50.943381+0000","last_active":"2024-05-08T01:36:53.601275+0000","last_peered":"2024-05-08T01:36:53.601275+0000","last_clean":"2024-05-08T01:36:53.601275+0000","last_became_active":"2024-05-08T01:13:50.942923+0000","last_became_peered":"2024-05-08T01:13:50.942923+0000","last_unstale":"2024-05-08T01:36:53.601275+0000","last_undegraded":"2024-05-08T01:36:53.601275+0000","last_fullsized":"2024-05-08T01:36:53.601275+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:43:23.433419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":303,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.600443+0000","last_change":"2024-05-08T01:13:50.946454+0000","last_active":"2024-05-08T01:36:53.600443+0000","last_peered":"2024-05-08T01:36:53.600443+0000","last_clean":"2024-05-08T01:36:53.600443+0000","last_became_active":"2024-05-08T01:13:50.945779+0000","last_became_peered":"2024-05-08T01:13:50.945779+0000","last_unstale":"2024-05-08T01:36:53.600443+0000","last_undegraded":"2024-05-08T01:36:53.600443+0000","last_fullsized":"2024-05-08T01:36:53.600443+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:13:29.335759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":303,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.597665+0000","last_change":"2024-05-08T01:13:50.945406+0000","last_active":"2024-05-08T01:36:53.597665+0000","last_peered":"2024-05-08T01:36:53.597665+0000","last_clean":"2024-05-08T01:36:53.597665+0000","last_became_active":"2024-05-08T01:13:50.944956+0000","last_became_peered":"2024-05-08T01:13:50.944956+0000","last_unstale":"2024-05-08T01:36:53.597665+0000","last_undegraded":"2024-05-08T01:36:53.597665+0000","last_fullsized":"2024-05-08T01:36:53.597665+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:06:46.842113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":301,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.426429+0000","last_change":"2024-05-08T01:13:50.946179+0000","last_active":"2024-05-08T01:36:52.426429+0000","last_peered":"2024-05-08T01:36:52.426429+0000","last_clean":"2024-05-08T01:36:52.426429+0000","last_became_active":"2024-05-08T01:13:50.945466+0000","last_became_peered":"2024-05-08T01:13:50.945466+0000","last_unstale":"2024-05-08T01:36:52.426429+0000","last_undegraded":"2024-05-08T01:36:52.426429+0000","last_fullsized":"2024-05-08T01:36:52.426429+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:26:28.180131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":301,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.411357+0000","last_change":"2024-05-08T01:13:50.945971+0000","last_active":"2024-05-08T01:36:52.411357+0000","last_peered":"2024-05-08T01:36:52.411357+0000","last_clean":"2024-05-08T01:36:52.411357+0000","last_became_active":"2024-05-08T01:13:50.945591+0000","last_became_peered":"2024-05-08T01:13:50.945591+0000","last_unstale":"2024-05-08T01:36:52.411357+0000","last_undegraded":"2024-05-08T01:36:52.411357+0000","last_fullsized":"2024-05-08T01:36:52.411357+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T07:47:08.586259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":303,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.603081+0000","last_change":"2024-05-08T01:13:50.942235+0000","last_active":"2024-05-08T01:36:53.603081+0000","last_peered":"2024-05-08T01:36:53.603081+0000","last_clean":"2024-05-08T01:36:53.603081+0000","last_became_active":"2024-05-08T01:13:50.941816+0000","last_became_peered":"2024-05-08T01:13:50.941816+0000","last_unstale":"2024-05-08T01:36:53.603081+0000","last_undegraded":"2024-05-08T01:36:53.603081+0000","last_fullsized":"2024-05-08T01:36:53.603081+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T13:00:29.849708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":436,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.607893+0000","last_change":"2024-05-08T01:13:45.924420+0000","last_active":"2024-05-08T01:36:53.607893+0000","last_peered":"2024-05-08T01:36:53.607893+0000","last_clean":"2024-05-08T01:36:53.607893+0000","last_became_active":"2024-05-08T01:13:45.921946+0000","last_became_peered":"2024-05-08T01:13:45.921946+0000","last_unstale":"2024-05-08T01:36:53.607893+0000","last_undegraded":"2024-05-08T01:36:53.607893+0000","last_fullsized":"2024-05-08T01:36:53.607893+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:44.905674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:44.905674+0000","last_clean_scrub_stamp":"2024-05-08T01:13:44.905674+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:33:37.631809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738656,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46360,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325480,"statfs":{"total":96636764160,"available":96589291520,"internally_reserved":0,"allocated":1323008,"data_stored":658311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738657,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41376,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330464,"statfs":{"total":96636764160,"available":96594395136,"internally_reserved":0,"allocated":1916928,"data_stored":1248679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738658,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46944,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324896,"statfs":{"total":96636764160,"available":96588693504,"internally_reserved":0,"allocated":1916928,"data_stored":1248679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}}
2024-05-08T01:37:30.701 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json
2024-05-08T01:37:30.994 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:37:30.995 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all
2024-05-08T01:37:31.015 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":865,"stamp":"2024-05-08T01:37:29.342503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":134680,"kb_used_data":5036,"kb_used_omap":4,"kb_used_meta":129595,"kb_avail":282980840,"statfs":{"total":289910292480,"available":289772380160,"internally_reserved":0,"allocated":5156864,"data_stored":3155669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":132705615},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002810"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":302,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.602190+0000","last_change":"2024-05-08T01:13:50.943750+0000","last_active":"2024-05-08T01:36:53.602190+0000","last_peered":"2024-05-08T01:36:53.602190+0000","last_clean":"2024-05-08T01:36:53.602190+0000","last_became_active":"2024-05-08T01:13:50.943014+0000","last_became_peered":"2024-05-08T01:13:50.943014+0000","last_unstale":"2024-05-08T01:36:53.602190+0000","last_undegraded":"2024-05-08T01:36:53.602190+0000","last_fullsized":"2024-05-08T01:36:53.602190+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:26:50.611238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":302,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.606876+0000","last_change":"2024-05-08T01:13:50.945953+0000","last_active":"2024-05-08T01:36:53.606876+0000","last_peered":"2024-05-08T01:36:53.606876+0000","last_clean":"2024-05-08T01:36:53.606876+0000","last_became_active":"2024-05-08T01:13:50.945427+0000","last_became_peered":"2024-05-08T01:13:50.945427+0000","last_unstale":"2024-05-08T01:36:53.606876+0000","last_undegraded":"2024-05-08T01:36:53.606876+0000","last_fullsized":"2024-05-08T01:36:53.606876+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:26:08.751583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":303,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.601275+0000","last_change":"2024-05-08T01:13:50.943381+0000","last_active":"2024-05-08T01:36:53.601275+0000","last_peered":"2024-05-08T01:36:53.601275+0000","last_clean":"2024-05-08T01:36:53.601275+0000","last_became_active":"2024-05-08T01:13:50.942923+0000","last_became_peered":"2024-05-08T01:13:50.942923+0000","last_unstale":"2024-05-08T01:36:53.601275+0000","last_undegraded":"2024-05-08T01:36:53.601275+0000","last_fullsized":"2024-05-08T01:36:53.601275+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:43:23.433419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":303,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.600443+0000","last_change":"2024-05-08T01:13:50.946454+0000","last_active":"2024-05-08T01:36:53.600443+0000","last_peered":"2024-05-08T01:36:53.600443+0000","last_clean":"2024-05-08T01:36:53.600443+0000","last_became_active":"2024-05-08T01:13:50.945779+0000","last_became_peered":"2024-05-08T01:13:50.945779+0000","last_unstale":"2024-05-08T01:36:53.600443+0000","last_undegraded":"2024-05-08T01:36:53.600443+0000","last_fullsized":"2024-05-08T01:36:53.600443+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:13:29.335759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":303,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.597665+0000","last_change":"2024-05-08T01:13:50.945406+0000","last_active":"2024-05-08T01:36:53.597665+0000","last_peered":"2024-05-08T01:36:53.597665+0000","last_clean":"2024-05-08T01:36:53.597665+0000","last_became_active":"2024-05-08T01:13:50.944956+0000","last_became_peered":"2024-05-08T01:13:50.944956+0000","last_unstale":"2024-05-08T01:36:53.597665+0000","last_undegraded":"2024-05-08T01:36:53.597665+0000","last_fullsized":"2024-05-08T01:36:53.597665+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:06:46.842113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":301,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.426429+0000","last_change":"2024-05-08T01:13:50.946179+0000","last_active":"2024-05-08T01:36:52.426429+0000","last_peered":"2024-05-08T01:36:52.426429+0000","last_clean":"2024-05-08T01:36:52.426429+0000","last_became_active":"2024-05-08T01:13:50.945466+0000","last_became_peered":"2024-05-08T01:13:50.945466+0000","last_unstale":"2024-05-08T01:36:52.426429+0000","last_undegraded":"2024-05-08T01:36:52.426429+0000","last_fullsized":"2024-05-08T01:36:52.426429+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:26:28.180131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":301,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.411357+0000","last_change":"2024-05-08T01:13:50.945971+0000","last_active":"2024-05-08T01:36:52.411357+0000","last_peered":"2024-05-08T01:36:52.411357+0000","last_clean":"2024-05-08T01:36:52.411357+0000","last_became_active":"2024-05-08T01:13:50.945591+0000","last_became_peered":"2024-05-08T01:13:50.945591+0000","last_unstale":"2024-05-08T01:36:52.411357+0000","last_undegraded":"2024-05-08T01:36:52.411357+0000","last_fullsized":"2024-05-08T01:36:52.411357+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T07:47:08.586259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":303,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.603081+0000","last_change":"2024-05-08T01:13:50.942235+0000","last_active":"2024-05-08T01:36:53.603081+0000","last_peered":"2024-05-08T01:36:53.603081+0000","last_clean":"2024-05-08T01:36:53.603081+0000","last_became_active":"2024-05-08T01:13:50.941816+0000","last_became_peered":"2024-05-08T01:13:50.941816+0000","last_unstale":"2024-05-08T01:36:53.603081+0000","last_undegraded":"2024-05-08T01:36:53.603081+0000","last_fullsized":"2024-05-08T01:36:53.603081+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T13:00:29.849708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":436,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.607893+0000","last_change":"2024-05-08T01:13:45.924420+0000","last_active":"2024-05-08T01:36:53.607893+0000","last_peered":"2024-05-08T01:36:53.607893+0000","last_clean":"2024-05-08T01:36:53.607893+0000","last_became_active":"2024-05-08T01:13:45.921946+0000","last_became_peered":"2024-05-08T01:13:45.921946+0000","last_unstale":"2024-05-08T01:36:53.607893+0000","last_undegraded":"2024-05-08T01:36:53.607893+0000","last_fullsized":"2024-05-08T01:36:53.607893+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:44.905674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:44.905674+0000","last_clean_scrub_stamp":"2024-05-08T01:13:44.905674+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:33:37.631809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738656,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46360,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325480,"statfs":{"total":96636764160,"available":96589291520,"internally_reserved":0,"allocated":1323008,"data_stored":658311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738657,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41376,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330464,"statfs":{"total":96636764160,"available":96594395136,"internally_reserved":0,"allocated":1916928,"data_stored":1248679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738658,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46944,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324896,"statfs":{"total":96636764160,"available":96588693504,"internally_reserved":0,"allocated":1916928,"data_stored":1248679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}}
2024-05-08T01:37:31.016 INFO:tasks.ceph.ceph_manager.cluster1:clean!
2024-05-08T01:37:31.016 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json
2024-05-08T01:37:31.301 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:37:31.301 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all
2024-05-08T01:37:31.318 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":865,"stamp":"2024-05-08T01:37:29.342503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":134680,"kb_used_data":5036,"kb_used_omap":4,"kb_used_meta":129595,"kb_avail":282980840,"statfs":{"total":289910292480,"available":289772380160,"internally_reserved":0,"allocated":5156864,"data_stored":3155669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":132705615},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002810"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":302,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.602190+0000","last_change":"2024-05-08T01:13:50.943750+0000","last_active":"2024-05-08T01:36:53.602190+0000","last_peered":"2024-05-08T01:36:53.602190+0000","last_clean":"2024-05-08T01:36:53.602190+0000","last_became_active":"2024-05-08T01:13:50.943014+0000","last_became_peered":"2024-05-08T01:13:50.943014+0000","last_unstale":"2024-05-08T01:36:53.602190+0000","last_undegraded":"2024-05-08T01:36:53.602190+0000","last_fullsized":"2024-05-08T01:36:53.602190+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:26:50.611238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":302,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.606876+0000","last_change":"2024-05-08T01:13:50.945953+0000","last_active":"2024-05-08T01:36:53.606876+0000","last_peered":"2024-05-08T01:36:53.606876+0000","last_clean":"2024-05-08T01:36:53.606876+0000","last_became_active":"2024-05-08T01:13:50.945427+0000","last_became_peered":"2024-05-08T01:13:50.945427+0000","last_unstale":"2024-05-08T01:36:53.606876+0000","last_undegraded":"2024-05-08T01:36:53.606876+0000","last_fullsized":"2024-05-08T01:36:53.606876+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:26:08.751583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":303,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.601275+0000","last_change":"2024-05-08T01:13:50.943381+0000","last_active":"2024-05-08T01:36:53.601275+0000","last_peered":"2024-05-08T01:36:53.601275+0000","last_clean":"2024-05-08T01:36:53.601275+0000","last_became_active":"2024-05-08T01:13:50.942923+0000","last_became_peered":"2024-05-08T01:13:50.942923+0000","last_unstale":"2024-05-08T01:36:53.601275+0000","last_undegraded":"2024-05-08T01:36:53.601275+0000","last_fullsized":"2024-05-08T01:36:53.601275+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:43:23.433419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":303,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.600443+0000","last_change":"2024-05-08T01:13:50.946454+0000","last_active":"2024-05-08T01:36:53.600443+0000","last_peered":"2024-05-08T01:36:53.600443+0000","last_clean":"2024-05-08T01:36:53.600443+0000","last_became_active":"2024-05-08T01:13:50.945779+0000","last_became_peered":"2024-05-08T01:13:50.945779+0000","last_unstale":"2024-05-08T01:36:53.600443+0000","last_undegraded":"2024-05-08T01:36:53.600443+0000","last_fullsized":"2024-05-08T01:36:53.600443+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:13:29.335759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":303,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.597665+0000","last_change":"2024-05-08T01:13:50.945406+0000","last_active":"2024-05-08T01:36:53.597665+0000","last_peered":"2024-05-08T01:36:53.597665+0000","last_clean":"2024-05-08T01:36:53.597665+0000","last_became_active":"2024-05-08T01:13:50.944956+0000","last_became_peered":"2024-05-08T01:13:50.944956+0000","last_unstale":"2024-05-08T01:36:53.597665+0000","last_undegraded":"2024-05-08T01:36:53.597665+0000","last_fullsized":"2024-05-08T01:36:53.597665+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:06:46.842113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":301,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.426429+0000","last_change":"2024-05-08T01:13:50.946179+0000","last_active":"2024-05-08T01:36:52.426429+0000","last_peered":"2024-05-08T01:36:52.426429+0000","last_clean":"2024-05-08T01:36:52.426429+0000","last_became_active":"2024-05-08T01:13:50.945466+0000","last_became_peered":"2024-05-08T01:13:50.945466+0000","last_unstale":"2024-05-08T01:36:52.426429+0000","last_undegraded":"2024-05-08T01:36:52.426429+0000","last_fullsized":"2024-05-08T01:36:52.426429+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:26:28.180131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":301,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.411357+0000","last_change":"2024-05-08T01:13:50.945971+0000","last_active":"2024-05-08T01:36:52.411357+0000","last_peered":"2024-05-08T01:36:52.411357+0000","last_clean":"2024-05-08T01:36:52.411357+0000","last_became_active":"2024-05-08T01:13:50.945591+0000","last_became_peered":"2024-05-08T01:13:50.945591+0000","last_unstale":"2024-05-08T01:36:52.411357+0000","last_undegraded":"2024-05-08T01:36:52.411357+0000","last_fullsized":"2024-05-08T01:36:52.411357+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T07:47:08.586259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":303,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.603081+0000","last_change":"2024-05-08T01:13:50.942235+0000","last_active":"2024-05-08T01:36:53.603081+0000","last_peered":"2024-05-08T01:36:53.603081+0000","last_clean":"2024-05-08T01:36:53.603081+0000","last_became_active":"2024-05-08T01:13:50.941816+0000","last_became_peered":"2024-05-08T01:13:50.941816+0000","last_unstale":"2024-05-08T01:36:53.603081+0000","last_undegraded":"2024-05-08T01:36:53.603081+0000","last_fullsized":"2024-05-08T01:36:53.603081+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T13:00:29.849708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":436,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.607893+0000","last_change":"2024-05-08T01:13:45.924420+0000","last_active":"2024-05-08T01:36:53.607893+0000","last_peered":"2024-05-08T01:36:53.607893+0000","last_clean":"2024-05-08T01:36:53.607893+0000","last_became_active":"2024-05-08T01:13:45.921946+0000","last_became_peered":"2024-05-08T01:13:45.921946+0000","last_unstale":"2024-05-08T01:36:53.607893+0000","last_undegraded":"2024-05-08T01:36:53.607893+0000","last_fullsized":"2024-05-08T01:36:53.607893+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:44.905674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:44.905674+0000","last_clean_scrub_stamp":"2024-05-08T01:13:44.905674+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:33:37.631809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738656,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46360,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325480,"statfs":{"total":96636764160,"available":96589291520,"internally_reserved":0,"allocated":1323008,"data_stored":658311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738657,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41376,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330464,"statfs":{"total":96636764160,"available":96594395136,"internally_reserved":0,"allocated":1916928,"data_stored":1248679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738658,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46944,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324896,"statfs":{"total":96636764160,"available":96588693504,"internally_reserved":0,"allocated":1916928,"data_stored":1248679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}}
2024-05-08T01:37:31.319 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json
2024-05-08T01:37:31.592 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:37:31.592 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":154,"fsid":"3284bad2-5fe9-40c3-bba2-9a50da63f434","created":"2024-05-08T01:13:34.190730+0000","modified":"2024-05-08T01:36:52.405125+0000","last_up_change":"2024-05-08T01:13:43.826579+0000","last_in_change":"2024-05-08T01:13:35.061884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T01:13:44.899677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-08T01:13:49.489312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3379a737-51ca-4783-a803-08467868290b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":3953308227},{"type":"v1","addr":"172.21.15.106:6817","nonce":3953308227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":3953308227},{"type":"v1","addr":"172.21.15.106:6819","nonce":3953308227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":3953308227},{"type":"v1","addr":"172.21.15.106:6823","nonce":3953308227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":3953308227},{"type":"v1","addr":"172.21.15.106:6821","nonce":3953308227}]},"public_addr":"172.21.15.106:6817/3953308227","cluster_addr":"172.21.15.106:6819/3953308227","heartbeat_back_addr":"172.21.15.106:6823/3953308227","heartbeat_front_addr":"172.21.15.106:6821/3953308227","state":["exists","up"]},{"osd":1,"uuid":"750e4933-f084-4144-96fc-82a2cd313ffc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":4128478887},{"type":"v1","addr":"172.21.15.106:6801","nonce":4128478887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":4128478887},{"type":"v1","addr":"172.21.15.106:6803","nonce":4128478887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":4128478887},{"type":"v1","addr":"172.21.15.106:6807","nonce":4128478887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":4128478887},{"type":"v1","addr":"172.21.15.106:6805","nonce":4128478887}]},"public_addr":"172.21.15.106:6801/4128478887","cluster_addr":"172.21.15.106:6803/4128478887","heartbeat_back_addr":"172.21.15.106:6807/4128478887","heartbeat_front_addr":"172.21.15.106:6805/4128478887","state":["exists","up"]},{"osd":2,"uuid":"8b237cc9-901b-4426-9a3f-a8c1b215f2a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":1358302753},{"type":"v1","addr":"172.21.15.106:6809","nonce":1358302753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":1358302753},{"type":"v1","addr":"172.21.15.106:6811","nonce":1358302753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":1358302753},{"type":"v1","addr":"172.21.15.106:6815","nonce":1358302753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":1358302753},{"type":"v1","addr":"172.21.15.106:6813","nonce":1358302753}]},"public_addr":"172.21.15.106:6809/1358302753","cluster_addr":"172.21.15.106:6811/1358302753","heartbeat_back_addr":"172.21.15.106:6815/1358302753","heartbeat_front_addr":"172.21.15.106:6813/1358302753","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:13:40.615948+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:13:40.595389+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T01:13:40.580330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":4,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
2024-05-08T01:37:32.609 INFO:tasks.ceph:Scrubbing cluster1.osd.0
2024-05-08T01:37:32.609 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0
2024-05-08T01:37:32.768 INFO:teuthology.orchestra.run.smithi106.stdout:{
2024-05-08T01:37:32.769 INFO:teuthology.orchestra.run.smithi106.stdout:    "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) "
2024-05-08T01:37:32.769 INFO:teuthology.orchestra.run.smithi106.stdout:}
2024-05-08T01:37:32.785 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0
2024-05-08T01:37:33.060 INFO:teuthology.orchestra.run.smithi106.stderr:instructed osd(s) 0 to deep-scrub
2024-05-08T01:37:33.077 INFO:tasks.ceph:Scrubbing cluster1.osd.1
2024-05-08T01:37:33.078 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0
2024-05-08T01:37:33.220 INFO:teuthology.orchestra.run.smithi106.stdout:{
2024-05-08T01:37:33.220 INFO:teuthology.orchestra.run.smithi106.stdout:    "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) "
2024-05-08T01:37:33.221 INFO:teuthology.orchestra.run.smithi106.stdout:}
2024-05-08T01:37:33.236 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1
2024-05-08T01:37:33.513 INFO:teuthology.orchestra.run.smithi106.stderr:instructed osd(s) 1 to deep-scrub
2024-05-08T01:37:33.530 INFO:tasks.ceph:Scrubbing cluster1.osd.2
2024-05-08T01:37:33.530 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0
2024-05-08T01:37:33.680 INFO:teuthology.orchestra.run.smithi106.stdout:{
2024-05-08T01:37:33.680 INFO:teuthology.orchestra.run.smithi106.stdout:    "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) "
2024-05-08T01:37:33.681 INFO:teuthology.orchestra.run.smithi106.stdout:}
2024-05-08T01:37:33.696 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2
2024-05-08T01:37:33.979 INFO:teuthology.orchestra.run.smithi106.stderr:instructed osd(s) 2 to deep-scrub
2024-05-08T01:37:33.996 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json
2024-05-08T01:37:34.272 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:37:34.272 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all
2024-05-08T01:37:34.290 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":867,"stamp":"2024-05-08T01:37:33.343331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":134680,"kb_used_data":5036,"kb_used_omap":4,"kb_used_meta":129595,"kb_avail":282980840,"statfs":{"total":289910292480,"available":289772380160,"internally_reserved":0,"allocated":5156864,"data_stored":3155669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":132705615},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002612"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":302,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.602190+0000","last_change":"2024-05-08T01:13:50.943750+0000","last_active":"2024-05-08T01:36:53.602190+0000","last_peered":"2024-05-08T01:36:53.602190+0000","last_clean":"2024-05-08T01:36:53.602190+0000","last_became_active":"2024-05-08T01:13:50.943014+0000","last_became_peered":"2024-05-08T01:13:50.943014+0000","last_unstale":"2024-05-08T01:36:53.602190+0000","last_undegraded":"2024-05-08T01:36:53.602190+0000","last_fullsized":"2024-05-08T01:36:53.602190+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:26:50.611238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":302,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.606876+0000","last_change":"2024-05-08T01:13:50.945953+0000","last_active":"2024-05-08T01:36:53.606876+0000","last_peered":"2024-05-08T01:36:53.606876+0000","last_clean":"2024-05-08T01:36:53.606876+0000","last_became_active":"2024-05-08T01:13:50.945427+0000","last_became_peered":"2024-05-08T01:13:50.945427+0000","last_unstale":"2024-05-08T01:36:53.606876+0000","last_undegraded":"2024-05-08T01:36:53.606876+0000","last_fullsized":"2024-05-08T01:36:53.606876+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:26:08.751583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":303,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.601275+0000","last_change":"2024-05-08T01:13:50.943381+0000","last_active":"2024-05-08T01:36:53.601275+0000","last_peered":"2024-05-08T01:36:53.601275+0000","last_clean":"2024-05-08T01:36:53.601275+0000","last_became_active":"2024-05-08T01:13:50.942923+0000","last_became_peered":"2024-05-08T01:13:50.942923+0000","last_unstale":"2024-05-08T01:36:53.601275+0000","last_undegraded":"2024-05-08T01:36:53.601275+0000","last_fullsized":"2024-05-08T01:36:53.601275+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:43:23.433419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":303,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.600443+0000","last_change":"2024-05-08T01:13:50.946454+0000","last_active":"2024-05-08T01:36:53.600443+0000","last_peered":"2024-05-08T01:36:53.600443+0000","last_clean":"2024-05-08T01:36:53.600443+0000","last_became_active":"2024-05-08T01:13:50.945779+0000","last_became_peered":"2024-05-08T01:13:50.945779+0000","last_unstale":"2024-05-08T01:36:53.600443+0000","last_undegraded":"2024-05-08T01:36:53.600443+0000","last_fullsized":"2024-05-08T01:36:53.600443+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:13:29.335759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":303,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.597665+0000","last_change":"2024-05-08T01:13:50.945406+0000","last_active":"2024-05-08T01:36:53.597665+0000","last_peered":"2024-05-08T01:36:53.597665+0000","last_clean":"2024-05-08T01:36:53.597665+0000","last_became_active":"2024-05-08T01:13:50.944956+0000","last_became_peered":"2024-05-08T01:13:50.944956+0000","last_unstale":"2024-05-08T01:36:53.597665+0000","last_undegraded":"2024-05-08T01:36:53.597665+0000","last_fullsized":"2024-05-08T01:36:53.597665+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:06:46.842113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":301,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.426429+0000","last_change":"2024-05-08T01:13:50.946179+0000","last_active":"2024-05-08T01:36:52.426429+0000","last_peered":"2024-05-08T01:36:52.426429+0000","last_clean":"2024-05-08T01:36:52.426429+0000","last_became_active":"2024-05-08T01:13:50.945466+0000","last_became_peered":"2024-05-08T01:13:50.945466+0000","last_unstale":"2024-05-08T01:36:52.426429+0000","last_undegraded":"2024-05-08T01:36:52.426429+0000","last_fullsized":"2024-05-08T01:36:52.426429+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:26:28.180131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":301,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:52.411357+0000","last_change":"2024-05-08T01:13:50.945971+0000","last_active":"2024-05-08T01:36:52.411357+0000","last_peered":"2024-05-08T01:36:52.411357+0000","last_clean":"2024-05-08T01:36:52.411357+0000","last_became_active":"2024-05-08T01:13:50.945591+0000","last_became_peered":"2024-05-08T01:13:50.945591+0000","last_unstale":"2024-05-08T01:36:52.411357+0000","last_undegraded":"2024-05-08T01:36:52.411357+0000","last_fullsized":"2024-05-08T01:36:52.411357+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T07:47:08.586259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":303,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.603081+0000","last_change":"2024-05-08T01:13:50.942235+0000","last_active":"2024-05-08T01:36:53.603081+0000","last_peered":"2024-05-08T01:36:53.603081+0000","last_clean":"2024-05-08T01:36:53.603081+0000","last_became_active":"2024-05-08T01:13:50.941816+0000","last_became_peered":"2024-05-08T01:13:50.941816+0000","last_unstale":"2024-05-08T01:36:53.603081+0000","last_undegraded":"2024-05-08T01:36:53.603081+0000","last_fullsized":"2024-05-08T01:36:53.603081+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:49.922637+0000","last_clean_scrub_stamp":"2024-05-08T01:13:49.922637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T13:00:29.849708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":436,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:36:53.607893+0000","last_change":"2024-05-08T01:13:45.924420+0000","last_active":"2024-05-08T01:36:53.607893+0000","last_peered":"2024-05-08T01:36:53.607893+0000","last_clean":"2024-05-08T01:36:53.607893+0000","last_became_active":"2024-05-08T01:13:45.921946+0000","last_became_peered":"2024-05-08T01:13:45.921946+0000","last_unstale":"2024-05-08T01:36:53.607893+0000","last_undegraded":"2024-05-08T01:36:53.607893+0000","last_fullsized":"2024-05-08T01:36:53.607893+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:13:44.905674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:13:44.905674+0000","last_clean_scrub_stamp":"2024-05-08T01:13:44.905674+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:33:37.631809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738656,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46360,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325480,"statfs":{"total":96636764160,"available":96589291520,"internally_reserved":0,"allocated":1323008,"data_stored":658311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738657,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41376,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330464,"statfs":{"total":96636764160,"available":96594395136,"internally_reserved":0,"allocated":1916928,"data_stored":1248679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738658,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46944,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324896,"statfs":{"total":96636764160,"available":96588693504,"internally_reserved":0,"allocated":1916928,"data_stored":1248679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}}
2024-05-08T01:37:34.291 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-05-08T01:13:49.922637+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=13, tm_sec=49, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=37, tm_sec=31, tm_wday=2, tm_yday=129, tm_isdst=0)
2024-05-08T01:37:34.291 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-05-08T01:13:49.922637+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=13, tm_sec=49, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=37, tm_sec=31, tm_wday=2, tm_yday=129, tm_isdst=0)
2024-05-08T01:37:34.291 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-05-08T01:13:49.922637+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=13, tm_sec=49, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=37, tm_sec=31, tm_wday=2, tm_yday=129, tm_isdst=0)
2024-05-08T01:37:34.291 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-05-08T01:13:49.922637+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=13, tm_sec=49, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=37, tm_sec=31, tm_wday=2, tm_yday=129, tm_isdst=0)
2024-05-08T01:37:34.291 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-05-08T01:13:49.922637+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=13, tm_sec=49, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=37, tm_sec=31, tm_wday=2, tm_yday=129, tm_isdst=0)
2024-05-08T01:37:34.291 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-05-08T01:13:49.922637+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=13, tm_sec=49, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=37, tm_sec=31, tm_wday=2, tm_yday=129, tm_isdst=0)
2024-05-08T01:37:34.291 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-05-08T01:13:49.922637+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=13, tm_sec=49, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=37, tm_sec=31, tm_wday=2, tm_yday=129, tm_isdst=0)
2024-05-08T01:37:34.292 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-05-08T01:13:49.922637+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=13, tm_sec=49, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=37, tm_sec=31, tm_wday=2, tm_yday=129, tm_isdst=0)
2024-05-08T01:37:34.292 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-05-08T01:13:44.905674+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=13, tm_sec=44, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=37, tm_sec=31, tm_wday=2, tm_yday=129, tm_isdst=0)
2024-05-08T01:37:34.292 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed.
2024-05-08T01:37:54.293 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json
2024-05-08T01:37:54.596 INFO:teuthology.orchestra.run.smithi106.stdout:
2024-05-08T01:37:54.597 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all
2024-05-08T01:37:54.614 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":877,"stamp":"2024-05-08T01:37:53.348266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":134660,"kb_used_data":5036,"kb_used_omap":4,"kb_used_meta":129595,"kb_avail":282980860,"statfs":{"total":289910292480,"available":289772400640,"internally_reserved":0,"allocated":5156864,"data_stored":3155669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":132705615},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003047"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":312,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:37:36.726212+0000","last_change":"2024-05-08T01:37:36.726212+0000","last_active":"2024-05-08T01:37:36.726212+0000","last_peered":"2024-05-08T01:37:36.726212+0000","last_clean":"2024-05-08T01:37:36.726212+0000","last_became_active":"2024-05-08T01:13:50.943014+0000","last_became_peered":"2024-05-08T01:13:50.943014+0000","last_unstale":"2024-05-08T01:37:36.726212+0000","last_undegraded":"2024-05-08T01:37:36.726212+0000","last_fullsized":"2024-05-08T01:37:36.726212+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:37:36.725941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:37:36.725941+0000","last_clean_scrub_stamp":"2024-05-08T01:37:36.725941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:36:21.159195+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":312,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:37:37.737506+0000","last_change":"2024-05-08T01:37:37.737506+0000","last_active":"2024-05-08T01:37:37.737506+0000","last_peered":"2024-05-08T01:37:37.737506+0000","last_clean":"2024-05-08T01:37:37.737506+0000","last_became_active":"2024-05-08T01:13:50.945427+0000","last_became_peered":"2024-05-08T01:13:50.945427+0000","last_unstale":"2024-05-08T01:37:37.737506+0000","last_undegraded":"2024-05-08T01:37:37.737506+0000","last_fullsized":"2024-05-08T01:37:37.737506+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:37:37.737136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:37:37.737136+0000","last_clean_scrub_stamp":"2024-05-08T01:37:37.737136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:27:19.886130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":312,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:37:35.712732+0000","last_change":"2024-05-08T01:37:35.712880+0000","last_active":"2024-05-08T01:37:35.712732+0000","last_peered":"2024-05-08T01:37:35.712732+0000","last_clean":"2024-05-08T01:37:35.712732+0000","last_became_active":"2024-05-08T01:13:50.942923+0000","last_became_peered":"2024-05-08T01:13:50.942923+0000","last_unstale":"2024-05-08T01:37:35.712732+0000","last_undegraded":"2024-05-08T01:37:35.712732+0000","last_fullsized":"2024-05-08T01:37:35.712732+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:37:35.712491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:37:35.712491+0000","last_clean_scrub_stamp":"2024-05-08T01:37:35.712491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T09:01:14.722315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":312,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:37:38.729293+0000","last_change":"2024-05-08T01:37:38.729347+0000","last_active":"2024-05-08T01:37:38.729293+0000","last_peered":"2024-05-08T01:37:38.729293+0000","last_clean":"2024-05-08T01:37:38.729293+0000","last_became_active":"2024-05-08T01:13:50.945779+0000","last_became_peered":"2024-05-08T01:13:50.945779+0000","last_unstale":"2024-05-08T01:37:38.729293+0000","last_undegraded":"2024-05-08T01:37:38.729293+0000","last_fullsized":"2024-05-08T01:37:38.729293+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:37:38.729085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:37:38.729085+0000","last_clean_scrub_stamp":"2024-05-08T01:37:38.729085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T07:44:09.323496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":312,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:37:33.329867+0000","last_change":"2024-05-08T01:37:33.329894+0000","last_active":"2024-05-08T01:37:33.329867+0000","last_peered":"2024-05-08T01:37:33.329867+0000","last_clean":"2024-05-08T01:37:33.329867+0000","last_became_active":"2024-05-08T01:13:50.944956+0000","last_became_peered":"2024-05-08T01:13:50.944956+0000","last_unstale":"2024-05-08T01:37:33.329867+0000","last_undegraded":"2024-05-08T01:37:33.329867+0000","last_fullsized":"2024-05-08T01:37:33.329867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:37:33.329768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:37:33.329768+0000","last_clean_scrub_stamp":"2024-05-08T01:37:33.329768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:33:15.528261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":310,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:37:35.793739+0000","last_change":"2024-05-08T01:37:35.793794+0000","last_active":"2024-05-08T01:37:35.793739+0000","last_peered":"2024-05-08T01:37:35.793739+0000","last_clean":"2024-05-08T01:37:35.793739+0000","last_became_active":"2024-05-08T01:13:50.945466+0000","last_became_peered":"2024-05-08T01:13:50.945466+0000","last_unstale":"2024-05-08T01:37:35.793739+0000","last_undegraded":"2024-05-08T01:37:35.793739+0000","last_fullsized":"2024-05-08T01:37:35.793739+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:37:35.793539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:37:35.793539+0000","last_clean_scrub_stamp":"2024-05-08T01:37:35.793539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:39:24.577015+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":310,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:37:34.801595+0000","last_change":"2024-05-08T01:37:34.801653+0000","last_active":"2024-05-08T01:37:34.801595+0000","last_peered":"2024-05-08T01:37:34.801595+0000","last_clean":"2024-05-08T01:37:34.801595+0000","last_became_active":"2024-05-08T01:13:50.945591+0000","last_became_peered":"2024-05-08T01:13:50.945591+0000","last_unstale":"2024-05-08T01:37:34.801595+0000","last_undegraded":"2024-05-08T01:37:34.801595+0000","last_fullsized":"2024-05-08T01:37:34.801595+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:37:34.801398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:37:34.801398+0000","last_clean_scrub_stamp":"2024-05-08T01:37:34.801398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T01:56:52.727586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":312,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:37:34.715007+0000","last_change":"2024-05-08T01:37:34.715060+0000","last_active":"2024-05-08T01:37:34.715007+0000","last_peered":"2024-05-08T01:37:34.715007+0000","last_clean":"2024-05-08T01:37:34.715007+0000","last_became_active":"2024-05-08T01:13:50.941816+0000","last_became_peered":"2024-05-08T01:13:50.941816+0000","last_unstale":"2024-05-08T01:37:34.715007+0000","last_undegraded":"2024-05-08T01:37:34.715007+0000","last_fullsized":"2024-05-08T01:37:34.715007+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:37:34.714800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:37:34.714800+0000","last_clean_scrub_stamp":"2024-05-08T01:37:34.714800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:26:37.416184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"154'99","reported_seq":447,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-05-08T01:37:33.759359+0000","last_change":"2024-05-08T01:37:33.759389+0000","last_active":"2024-05-08T01:37:33.759359+0000","last_peered":"2024-05-08T01:37:33.759359+0000","last_clean":"2024-05-08T01:37:33.759359+0000","last_became_active":"2024-05-08T01:13:45.921946+0000","last_became_peered":"2024-05-08T01:13:45.921946+0000","last_unstale":"2024-05-08T01:37:33.759359+0000","last_undegraded":"2024-05-08T01:37:33.759359+0000","last_fullsized":"2024-05-08T01:37:33.759359+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'99","last_scrub_stamp":"2024-05-08T01:37:33.759276+0000","last_deep_scrub":"154'99","last_deep_scrub_stamp":"2024-05-08T01:37:33.759276+0000","last_clean_scrub_stamp":"2024-05-08T01:37:33.759276+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:45:59.654140+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738660,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46356,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325484,"statfs":{"total":96636764160,"available":96589295616,"internally_reserved":0,"allocated":1323008,"data_stored":658311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738661,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41368,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330472,"statfs":{"total":96636764160,"available":96594403328,"internally_reserved":0,"allocated":1916928,"data_stored":1248679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738662,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46936,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324904,"statfs":{"total":96636764160,"available":96588701696,"internally_reserved":0,"allocated":1916928,"data_stored":1248679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}}
2024-05-08T01:37:54.617 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false
2024-05-08T01:37:54.938 INFO:teuthology.misc:Shutting down mds daemons...
2024-05-08T01:37:54.938 INFO:teuthology.misc:Shutting down osd daemons...
2024-05-08T01:37:54.938 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit
2024-05-08T01:37:54.938 INFO:teuthology.orchestra.run:waiting for 300
2024-05-08T01:37:55.027 INFO:tasks.ceph.cluster1.osd.0:Stopped
2024-05-08T01:37:55.027 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit
2024-05-08T01:37:55.027 INFO:teuthology.orchestra.run:waiting for 300
2024-05-08T01:37:55.119 INFO:tasks.ceph.cluster1.osd.1:Stopped
2024-05-08T01:37:55.120 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit
2024-05-08T01:37:55.120 INFO:teuthology.orchestra.run:waiting for 300
2024-05-08T01:37:55.213 INFO:tasks.ceph.cluster1.osd.2:Stopped
2024-05-08T01:37:55.213 INFO:teuthology.misc:Shutting down mgr daemons...
2024-05-08T01:37:55.214 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit
2024-05-08T01:37:55.214 INFO:teuthology.orchestra.run:waiting for 300
2024-05-08T01:37:55.243 INFO:tasks.ceph.cluster1.mgr.x:Stopped
2024-05-08T01:37:55.243 INFO:teuthology.misc:Shutting down mon daemons...
2024-05-08T01:37:55.243 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit
2024-05-08T01:37:55.244 INFO:teuthology.orchestra.run:waiting for 300
2024-05-08T01:37:55.304 INFO:tasks.ceph.cluster1.mon.a:Stopped
2024-05-08T01:37:55.304 INFO:tasks.ceph:Checking cluster log for badness...
2024-05-08T01:37:55.304 DEBUG:teuthology.orchestra.run.smithi106:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1
2024-05-08T01:37:55.359 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi106.front.sepia.ceph.com
2024-05-08T01:37:55.359 DEBUG:teuthology.orchestra.run.smithi106:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0
2024-05-08T01:37:55.638 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi106.front.sepia.ceph.com
2024-05-08T01:37:55.638 DEBUG:teuthology.orchestra.run.smithi106:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1
2024-05-08T01:37:55.846 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi106.front.sepia.ceph.com
2024-05-08T01:37:55.846 DEBUG:teuthology.orchestra.run.smithi106:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2
2024-05-08T01:37:56.030 INFO:tasks.ceph:Archiving mon data...
2024-05-08T01:37:56.030 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696665/data/cluster1.mon.a.tgz
2024-05-08T01:37:56.031 DEBUG:teuthology.orchestra.run.smithi106:> mktemp
2024-05-08T01:37:56.036 INFO:teuthology.orchestra.run.smithi106.stdout:/tmp/tmp.VLrrMBIob1
2024-05-08T01:37:56.036 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.VLrrMBIob1
2024-05-08T01:37:56.187 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0666 /tmp/tmp.VLrrMBIob1
2024-05-08T01:37:56.248 DEBUG:teuthology.orchestra.remote:smithi106:/tmp/tmp.VLrrMBIob1 is 446KB
2024-05-08T01:37:56.309 DEBUG:teuthology.orchestra.run.smithi106:> rm -fr /tmp/tmp.VLrrMBIob1
2024-05-08T01:37:56.314 INFO:tasks.ceph:Cleaning ceph cluster...
2024-05-08T01:37:56.314 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid
2024-05-08T01:37:56.360 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid
2024-05-08T01:37:56.425 INFO:teuthology.util.scanner:summary_data or yaml_file is empty!
2024-05-08T01:37:56.431 INFO:teuthology.util.scanner:summary_data or yaml_file is empty!
2024-05-08T01:37:56.431 INFO:tasks.ceph:Archiving crash dumps...
2024-05-08T01:37:56.432 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696665/remote/smithi106/crash
2024-05-08T01:37:56.433 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/lib/ceph/crash -- .
2024-05-08T01:37:56.480 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696665/remote/smithi142/crash
2024-05-08T01:37:56.481 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /var/lib/ceph/crash -- .
2024-05-08T01:37:56.494 INFO:tasks.ceph:Compressing logs...
2024-05-08T01:37:56.494 DEBUG:teuthology.orchestra.run.smithi106:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --
2024-05-08T01:37:56.528 DEBUG:teuthology.orchestra.run.smithi142:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --
2024-05-08T01:37:56.544 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29458.log
2024-05-08T01:37:56.545 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28499.log
2024-05-08T01:37:56.547 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.29458.log:	gzip  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29458.log.gz -5
2024-05-08T01:37:56.547 INFO:teuthology.orchestra.run.smithi106.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.24078.log
2024-05-08T01:37:56.547 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.28499.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28499.log.gz
2024-05-08T01:37:56.547 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28949.log
2024-05-08T01:37:56.547 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.24078.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28871.log
2024-05-08T01:37:56.548 INFO:teuthology.orchestra.run.smithi106.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24078.log.gz
2024-05-08T01:37:56.548 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.28949.log:	gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log
2024-05-08T01:37:56.548 INFO:teuthology.orchestra.run.smithi106.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28949.log.gz
2024-05-08T01:37:56.548 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.28871.log:	gzip  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28871.log.gz
2024-05-08T01:37:56.548 INFO:teuthology.orchestra.run.smithi106.stderr: -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20338.log
2024-05-08T01:37:56.551 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1.audit.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28723.log
2024-05-08T01:37:56.551 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph.tmp-client.admin.20338.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28875.log
2024-05-08T01:37:56.552 INFO:teuthology.orchestra.run.smithi106.stderr:  0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20338.log.gz
2024-05-08T01:37:56.552 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.28723.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28723.log.gz
2024-05-08T01:37:56.552 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log
2024-05-08T01:37:56.552 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.28875.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28875.log.gz
2024-05-08T01:37:56.552 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28611.log
2024-05-08T01:37:56.552 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-mon.a.log:	 /var/log/ceph/cluster1-client.admin.29380.log
2024-05-08T01:37:56.552 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.28611.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28611.log.gz
2024-05-08T01:37:56.552 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28799.log
2024-05-08T01:37:56.552 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.29380.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28975.log
2024-05-08T01:37:56.552 INFO:teuthology.orchestra.run.smithi106.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29380.log.gz
2024-05-08T01:37:56.552 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.28799.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28799.log.gz
2024-05-08T01:37:56.552 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28585.log
2024-05-08T01:37:56.552 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.28975.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log
2024-05-08T01:37:56.552 INFO:teuthology.orchestra.run.smithi106.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28975.log.gz
2024-05-08T01:37:56.553 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.28585.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28585.log.gz
2024-05-08T01:37:56.553 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24113.log
2024-05-08T01:37:56.553 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster2-mgr.x.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29302.log
2024-05-08T01:37:56.555 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.24113.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24113.log.gz
2024-05-08T01:37:56.555 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log
2024-05-08T01:37:56.555 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28663.log
2024-05-08T01:37:56.555 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.29302.log:	/var/log/ceph/cluster1-mgr.x.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29302.log.gz
2024-05-08T01:37:56.555 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29510.log
2024-05-08T01:37:56.555 INFO:teuthology.orchestra.run.smithi106.stderr:gzip/var/log/ceph/cluster1-client.admin.28663.log:	 -5 --verbose -- /var/log/ceph/cluster1-osd.2.log
2024-05-08T01:37:56.555 INFO:teuthology.orchestra.run.smithi106.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28663.log.gz
2024-05-08T01:37:56.555 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.29510.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log
2024-05-08T01:37:56.555 INFO:teuthology.orchestra.run.smithi106.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29510.log.gz
2024-05-08T01:37:56.555 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-osd.2.log:	 /var/log/ceph/cluster1-client.admin.28718.log
2024-05-08T01:37:56.555 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-osd.0.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24175.log
2024-05-08T01:37:56.560 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.28718.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29001.log
2024-05-08T01:37:56.560 INFO:teuthology.orchestra.run.smithi106.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28718.log.gz
2024-05-08T01:37:56.560 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.24175.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24175.log.gz
2024-05-08T01:37:56.560 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29328.log
2024-05-08T01:37:56.561 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.29001.log:	  0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29406.log
2024-05-08T01:37:56.561 INFO:teuthology.orchestra.run.smithi106.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.29001.log.gz
2024-05-08T01:37:56.561 INFO:teuthology.orchestra.run.smithi106.stderr: 93.6%/var/log/ceph/cluster1-client.admin.29328.log:	 -- replaced with /var/log/ceph/cluster1.audit.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28689.log
2024-05-08T01:37:56.561 INFO:teuthology.orchestra.run.smithi106.stderr:
2024-05-08T01:37:56.561 INFO:teuthology.orchestra.run.smithi106.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29328.log.gz
2024-05-08T01:37:56.561 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.29406.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29406.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29250.log
2024-05-08T01:37:56.561 INFO:teuthology.orchestra.run.smithi106.stderr:
2024-05-08T01:37:56.561 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.28689.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29534.log
2024-05-08T01:37:56.561 INFO:teuthology.orchestra.run.smithi106.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28689.log.gz
2024-05-08T01:37:56.561 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29354.log
2024-05-08T01:37:56.561 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.29250.log:	/var/log/ceph/cluster1-client.admin.29534.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28801.log
2024-05-08T01:37:56.561 INFO:teuthology.orchestra.run.smithi106.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29534.log.gz
2024-05-08T01:37:56.561 INFO:teuthology.orchestra.run.smithi106.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29250.log.gz
2024-05-08T01:37:56.562 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.29354.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29354.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24054.log
2024-05-08T01:37:56.562 INFO:teuthology.orchestra.run.smithi106.stderr:
2024-05-08T01:37:56.562 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.28801.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28637.log
2024-05-08T01:37:56.562 INFO:teuthology.orchestra.run.smithi106.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28801.log.gz
2024-05-08T01:37:56.571 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24030.log
2024-05-08T01:37:56.572 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.24054.log:	/var/log/ceph/cluster1-client.admin.28637.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29224.log
2024-05-08T01:37:56.572 INFO:teuthology.orchestra.run.smithi106.stderr:  0.0%  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24054.log.gz
2024-05-08T01:37:56.572 INFO:teuthology.orchestra.run.smithi106.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.28637.log.gz
2024-05-08T01:37:56.572 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.24030.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24030.log.gz
2024-05-08T01:37:56.572 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29432.log
2024-05-08T01:37:56.572 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28879.log
2024-05-08T01:37:56.572 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.29224.log:	/var/log/ceph/cluster1-client.admin.29432.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28722.log
2024-05-08T01:37:56.572 INFO:teuthology.orchestra.run.smithi106.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29432.log.gz
2024-05-08T01:37:56.572 INFO:teuthology.orchestra.run.smithi106.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29224.log.gz
2024-05-08T01:37:56.572 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.28879.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28879.log.gz
2024-05-08T01:37:56.572 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log
2024-05-08T01:37:56.572 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.28722.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29484.log
2024-05-08T01:37:56.572 INFO:teuthology.orchestra.run.smithi106.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28722.log.gz
2024-05-08T01:37:56.573 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23936.log
2024-05-08T01:37:56.573 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.29484.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28793.log
2024-05-08T01:37:56.573 INFO:teuthology.orchestra.run.smithi106.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29484.log.gz
2024-05-08T01:37:56.573 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.23936.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.23936.log.gz
2024-05-08T01:37:56.573 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29276.log
2024-05-08T01:37:56.573 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.28793.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28560.log
2024-05-08T01:37:56.573 INFO:teuthology.orchestra.run.smithi106.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28793.log.gz
2024-05-08T01:37:56.573 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.29276.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29276.log.gz
2024-05-08T01:37:56.573 INFO:teuthology.orchestra.run.smithi106.stderr: 88.0% -- replaced with /var/log/ceph/cluster1.log.gz
2024-05-08T01:37:56.573 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log
2024-05-08T01:37:56.573 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-client.admin.28560.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28560.log.gz
2024-05-08T01:37:56.599 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33805.log
2024-05-08T01:37:56.599 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68555.log
2024-05-08T01:37:56.599 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.33805.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61170.log
2024-05-08T01:37:56.599 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.33805.log.gz
2024-05-08T01:37:56.599 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.68555.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41651.log
2024-05-08T01:37:56.599 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68555.log.gz
2024-05-08T01:37:56.599 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.69390.log
2024-05-08T01:37:56.599 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.61170.log:	/var/log/ceph/cluster1-client.mirror.41651.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60772.log
2024-05-08T01:37:56.600 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.69390.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62174.log
2024-05-08T01:37:56.600 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61170.log.gz
2024-05-08T01:37:56.600 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.69390.log.gz
2024-05-08T01:37:56.600 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41651.log.gz
2024-05-08T01:37:56.600 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.60772.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52519.log
2024-05-08T01:37:56.600 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.62174.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36587.log
2024-05-08T01:37:56.600 INFO:teuthology.orchestra.run.smithi142.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.admin.60772.log.gz
2024-05-08T01:37:56.600 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.52519.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32631.log
2024-05-08T01:37:56.600 INFO:teuthology.orchestra.run.smithi142.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62174.log.gz
2024-05-08T01:37:56.600 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.36587.log:	 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52519.log.gz
2024-05-08T01:37:56.600 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29972.log
2024-05-08T01:37:56.600 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.36587.log.gz
2024-05-08T01:37:56.601 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.32631.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48921.log
2024-05-08T01:37:56.601 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.29972.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38351.log
2024-05-08T01:37:56.601 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.32631.log.gz
2024-05-08T01:37:56.601 INFO:teuthology.orchestra.run.smithi142.stderr: 64.1% -- replaced with /var/log/ceph/cluster1-client.mirror.29972.log.gz
2024-05-08T01:37:56.601 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.48921.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51244.log
2024-05-08T01:37:56.601 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.48921.log.gz
2024-05-08T01:37:56.601 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.38351.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34013.log
2024-05-08T01:37:56.601 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.51244.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34233.log
2024-05-08T01:37:56.601 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.34013.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32299.log
2024-05-08T01:37:56.601 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.51244.log.gz
2024-05-08T01:37:56.601 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.34233.log:	 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.34013.log.gz
2024-05-08T01:37:56.601 INFO:teuthology.orchestra.run.smithi142.stderr:gzip 91.4% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.38351.log.gz --verbose
2024-05-08T01:37:56.601 INFO:teuthology.orchestra.run.smithi142.stderr: -- /var/log/ceph/cluster2-client.admin.50356.log
2024-05-08T01:37:56.601 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.32299.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.34233.log.gz
2024-05-08T01:37:56.602 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.30406.log
2024-05-08T01:37:56.602 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.50356.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.32299.log.gz
2024-05-08T01:37:56.602 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72507.log
2024-05-08T01:37:56.602 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.3.30406.log:	 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.50356.log.gz
2024-05-08T01:37:56.602 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60422.log
2024-05-08T01:37:56.602 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.72507.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35339.log
2024-05-08T01:37:56.602 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72507.log.gz
2024-05-08T01:37:56.602 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.60422.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69180.log
2024-05-08T01:37:56.602 INFO:teuthology.orchestra.run.smithi142.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60422.log.gz
2024-05-08T01:37:56.602 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.35339.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62750.log
2024-05-08T01:37:56.602 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.69180.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45375.log
2024-05-08T01:37:56.602 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35339.log.gz
2024-05-08T01:37:56.602 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.62750.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66060.log
2024-05-08T01:37:56.603 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.45375.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62750.log.gz
2024-05-08T01:37:56.603 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67255.log
2024-05-08T01:37:56.603 INFO:teuthology.orchestra.run.smithi142.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69180.log.gz
2024-05-08T01:37:56.603 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.45375.log.gz
2024-05-08T01:37:56.603 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57778.log
2024-05-08T01:37:56.603 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.66060.log:	/var/log/ceph/cluster1-client.mirror.67255.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50662.log
2024-05-08T01:37:56.603 INFO:teuthology.orchestra.run.smithi142.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67255.log.gz
2024-05-08T01:37:56.603 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.57778.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55271.log
2024-05-08T01:37:56.603 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.57778.log.gz
2024-05-08T01:37:56.603 INFO:teuthology.orchestra.run.smithi142.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.3.30406.log.gz
2024-05-08T01:37:56.603 INFO:teuthology.orchestra.run.smithi142.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66060.log.gz
2024-05-08T01:37:56.603 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53007.log
2024-05-08T01:37:56.603 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.50662.log:	/var/log/ceph/cluster2-client.admin.55271.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36446.log
2024-05-08T01:37:56.603 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.50662.log.gz
2024-05-08T01:37:56.603 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.53007.log:	 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.55271.log.gz
2024-05-08T01:37:56.604 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53520.log
2024-05-08T01:37:56.604 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.53007.log.gz
2024-05-08T01:37:56.604 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.36446.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58175.log
2024-05-08T01:37:56.604 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.53520.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52853.log
2024-05-08T01:37:56.604 INFO:teuthology.orchestra.run.smithi142.stderr: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36446.log.gz
2024-05-08T01:37:56.604 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.53520.log.gz
2024-05-08T01:37:56.604 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73016.log
2024-05-08T01:37:56.604 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58175.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37664.log
2024-05-08T01:37:56.604 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.52853.log:	 83.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58175.log.gz
2024-05-08T01:37:56.604 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.73016.log:	 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.52853.log.gz
2024-05-08T01:37:56.604 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42922.log
2024-05-08T01:37:56.604 INFO:teuthology.orchestra.run.smithi142.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.73016.log.gz
2024-05-08T01:37:56.604 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.37664.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65972.log
2024-05-08T01:37:56.604 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61509.log
2024-05-08T01:37:56.605 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.42922.log:	/var/log/ceph/cluster2-client.mirror.65972.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63648.log
2024-05-08T01:37:56.605 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.37664.log.gz
2024-05-08T01:37:56.605 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.61509.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.42922.log.gz
2024-05-08T01:37:56.605 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71018.log
2024-05-08T01:37:56.605 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65972.log.gz
2024-05-08T01:37:56.605 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.63648.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29378.log
2024-05-08T01:37:56.605 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61509.log.gz
2024-05-08T01:37:56.605 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.71018.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46727.log
2024-05-08T01:37:56.605 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71018.log.gz
2024-05-08T01:37:56.605 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.29378.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29378.log.gz
2024-05-08T01:37:56.605 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39324.log
2024-05-08T01:37:56.605 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.46727.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50934.log
2024-05-08T01:37:56.605 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46727.log.gz
2024-05-08T01:37:56.606 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.39324.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47577.log
2024-05-08T01:37:56.606 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.39324.log.gz
2024-05-08T01:37:56.606 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.50934.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42046.log
2024-05-08T01:37:56.606 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.50934.log.gz
2024-05-08T01:37:56.606 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71407.log
2024-05-08T01:37:56.606 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.47577.log:	/var/log/ceph/cluster1-client.admin.42046.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30497.log
2024-05-08T01:37:56.606 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.42046.log.gz
2024-05-08T01:37:56.606 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.71407.log:	 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47577.log.gz
2024-05-08T01:37:56.606 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74755.log
2024-05-08T01:37:56.606 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71407.log.gz
2024-05-08T01:37:56.606 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.30497.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30023.log
2024-05-08T01:37:56.606 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34075.log
2024-05-08T01:37:56.606 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.74755.log:	 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.30497.log.gz
2024-05-08T01:37:56.606 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74755.log.gz
2024-05-08T01:37:56.607 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.30023.log:	 56.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30023.log.gz
2024-05-08T01:37:56.607 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40864.log
2024-05-08T01:37:56.607 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.34075.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70048.log
2024-05-08T01:37:56.607 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.40864.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29703.log
2024-05-08T01:37:56.607 INFO:teuthology.orchestra.run.smithi142.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.admin.34075.log.gz
2024-05-08T01:37:56.607 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.40864.log.gz
2024-05-08T01:37:56.607 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.70048.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47667.log
2024-05-08T01:37:56.607 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.29703.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50598.log
2024-05-08T01:37:56.607 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29703.log.gz
2024-05-08T01:37:56.607 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70048.log.gz
2024-05-08T01:37:56.607 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.47667.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67278.log
2024-05-08T01:37:56.607 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47667.log.gz
2024-05-08T01:37:56.607 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.50598.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45984.log
2024-05-08T01:37:56.607 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.67278.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29173.log
2024-05-08T01:37:56.607 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50598.log.gz
2024-05-08T01:37:56.608 INFO:teuthology.orchestra.run.smithi142.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67278.log.gz
2024-05-08T01:37:56.608 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.45984.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32387.log
2024-05-08T01:37:56.608 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.45984.log.gz
2024-05-08T01:37:56.608 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.29173.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36145.log
2024-05-08T01:37:56.608 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29173.log.gz
2024-05-08T01:37:56.608 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.32387.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70344.log
2024-05-08T01:37:56.608 INFO:teuthology.orchestra.run.smithi142.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63648.log.gz
2024-05-08T01:37:56.608 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.36145.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45858.log
2024-05-08T01:37:56.608 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.70344.log:	 -- replaced with /var/log/ceph/cluster1-client.admin.36145.log.gz
2024-05-08T01:37:56.608 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45890.log
2024-05-08T01:37:56.608 INFO:teuthology.orchestra.run.smithi142.stderr:gzip/var/log/ceph/cluster2-client.mirror.45858.log:	 -5 --verbose -- /var/log/ceph/cluster1-client.admin.46568.log
2024-05-08T01:37:56.608 INFO:teuthology.orchestra.run.smithi142.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32387.log.gz
2024-05-08T01:37:56.608 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.45890.log:	 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45858.log.gz
2024-05-08T01:37:56.609 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44819.log
2024-05-08T01:37:56.609 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.45890.log.gz
2024-05-08T01:37:56.609 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47824.log
2024-05-08T01:37:56.609 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.46568.log:	/var/log/ceph/cluster2-client.mirror.44819.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28835.log
2024-05-08T01:37:56.609 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44819.log.gz
2024-05-08T01:37:56.609 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34354.log
2024-05-08T01:37:56.609 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.47824.log:	 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.46568.log.gz
2024-05-08T01:37:56.609 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.28835.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68497.log
2024-05-08T01:37:56.609 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28835.log.gz
2024-05-08T01:37:56.609 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.47824.log.gz
2024-05-08T01:37:56.609 INFO:teuthology.orchestra.run.smithi142.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70344.log.gz
2024-05-08T01:37:56.609 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.34354.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32367.log
2024-05-08T01:37:56.610 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.68497.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45647.log
2024-05-08T01:37:56.610 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.34354.log.gz
2024-05-08T01:37:56.610 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60294.log
2024-05-08T01:37:56.610 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.32367.log:	 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.68497.log.gz
2024-05-08T01:37:56.610 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.45647.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49284.log
2024-05-08T01:37:56.610 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.60294.log:	 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.45647.log.gz
2024-05-08T01:37:56.610 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50266.log
2024-05-08T01:37:56.610 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60294.log.gz
2024-05-08T01:37:56.610 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40947.log
2024-05-08T01:37:56.610 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.49284.log:	/var/log/ceph/cluster1-client.admin.50266.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28947.log
2024-05-08T01:37:56.610 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.49284.log.gz
2024-05-08T01:37:56.610 INFO:teuthology.orchestra.run.smithi142.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32367.log.gz
2024-05-08T01:37:56.610 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.40947.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.50266.log.gz
2024-05-08T01:37:56.611 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45955.log
2024-05-08T01:37:56.611 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40947.log.gz
2024-05-08T01:37:56.611 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.28947.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29277.log
2024-05-08T01:37:56.611 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28947.log.gz
2024-05-08T01:37:56.611 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.35334.log
2024-05-08T01:37:56.611 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.45955.log:	gzip -5 --verbose --/var/log/ceph/cluster2-client.admin.29277.log:	 /var/log/ceph/cluster2-client.admin.46798.log
2024-05-08T01:37:56.611 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29277.log.gz
2024-05-08T01:37:56.611 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.45955.log.gz
2024-05-08T01:37:56.611 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.3.35334.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73431.log
2024-05-08T01:37:56.611 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.46798.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62418.log
2024-05-08T01:37:56.611 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.73431.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74025.log
2024-05-08T01:37:56.611 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.46798.log.gz
2024-05-08T01:37:56.611 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.62418.log:	 86.7% -- replaced with /var/log/ceph/cluster1-client.admin.73431.log.gz
2024-05-08T01:37:56.612 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49138.log
2024-05-08T01:37:56.612 INFO:teuthology.orchestra.run.smithi142.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62418.log.gz
2024-05-08T01:37:56.612 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.74025.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58949.log
2024-05-08T01:37:56.612 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74025.log.gz
2024-05-08T01:37:56.612 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.49138.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64288.log
2024-05-08T01:37:56.612 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58949.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40368.log
2024-05-08T01:37:56.612 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.49138.log.gz
2024-05-08T01:37:56.612 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.64288.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51917.log
2024-05-08T01:37:56.612 INFO:teuthology.orchestra.run.smithi142.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.58949.log.gz
2024-05-08T01:37:56.612 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.40368.log:	 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64288.log.gz
2024-05-08T01:37:56.612 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40686.log
2024-05-08T01:37:56.612 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.51917.log:	 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.40368.log.gz
2024-05-08T01:37:56.612 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33677.log
2024-05-08T01:37:56.613 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.51917.log.gz
2024-05-08T01:37:56.613 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.40686.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50962.log
2024-05-08T01:37:56.613 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.33677.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60608.log
2024-05-08T01:37:56.613 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.50962.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74677.log
2024-05-08T01:37:56.613 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40686.log.gz
2024-05-08T01:37:56.613 INFO:teuthology.orchestra.run.smithi142.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.admin.33677.log.gz
2024-05-08T01:37:56.613 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.60608.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53877.log
2024-05-08T01:37:56.613 INFO:teuthology.orchestra.run.smithi142.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50962.log.gz
2024-05-08T01:37:56.613 INFO:teuthology.orchestra.run.smithi142.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60608.log.gz
2024-05-08T01:37:56.613 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.74677.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51676.log
2024-05-08T01:37:56.613 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74677.log.gz
2024-05-08T01:37:56.613 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.31439.log
2024-05-08T01:37:56.613 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.53877.log:	/var/log/ceph/cluster2-client.admin.51676.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69476.log
2024-05-08T01:37:56.613 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53877.log.gz
2024-05-08T01:37:56.614 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.51676.log.gz
2024-05-08T01:37:56.614 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.3.31439.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68226.log
2024-05-08T01:37:56.646 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49524.log
2024-05-08T01:37:56.647 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.69476.log:	/var/log/ceph/cluster1-client.mirror.68226.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64558.log
2024-05-08T01:37:56.647 INFO:teuthology.orchestra.run.smithi142.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68226.log.gz
2024-05-08T01:37:56.647 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.49524.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38482.log
2024-05-08T01:37:56.647 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.64558.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33346.log
2024-05-08T01:37:56.647 INFO:teuthology.orchestra.run.smithi142.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64558.log.gz
2024-05-08T01:37:56.647 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.49524.log.gz
2024-05-08T01:37:56.647 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67695.log
2024-05-08T01:37:56.647 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.38482.log:	/var/log/ceph/cluster1-client.admin.33346.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54583.log
2024-05-08T01:37:56.647 INFO:teuthology.orchestra.run.smithi142.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69476.log.gz
2024-05-08T01:37:56.647 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.38482.log.gz
2024-05-08T01:37:56.647 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58240.log
2024-05-08T01:37:56.647 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.67695.log:	 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.33346.log.gz
2024-05-08T01:37:56.647 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.54583.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72856.log
2024-05-08T01:37:56.648 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67695.log.gz
2024-05-08T01:37:56.648 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43116.log
2024-05-08T01:37:56.648 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58240.log:	 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.54583.log.gz
2024-05-08T01:37:56.648 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.72856.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35809.log
2024-05-08T01:37:56.648 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.72856.log.gz
2024-05-08T01:37:56.648 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.43116.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68522.log
2024-05-08T01:37:56.648 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62797.log
2024-05-08T01:37:56.648 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.35809.log:	/var/log/ceph/cluster2-client.mirror.68522.log:	 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.43116.log.gz
2024-05-08T01:37:56.648 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42893.log
2024-05-08T01:37:56.648 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.62797.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46481.log
2024-05-08T01:37:56.648 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62797.log.gz
2024-05-08T01:37:56.648 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.42893.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38801.log
2024-05-08T01:37:56.648 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.46481.log:	 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.42893.log.gz
2024-05-08T01:37:56.648 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46481.log.gz
2024-05-08T01:37:56.649 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38885.log
2024-05-08T01:37:56.649 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.38801.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.35279.log
2024-05-08T01:37:56.649 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8%/var/log/ceph/cluster1-client.admin.38885.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48890.log
2024-05-08T01:37:56.649 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.38801.log.gz
2024-05-08T01:37:56.649 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.3.35279.log:	 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.38885.log.gz
2024-05-08T01:37:56.649 INFO:teuthology.orchestra.run.smithi142.stderr: 20.5% -- replaced with /var/log/ceph/cluster2-client.mirror.3.35279.log.gz
2024-05-08T01:37:56.649 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63432.log
2024-05-08T01:37:56.649 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.48890.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32267.log
2024-05-08T01:37:56.649 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.63432.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44255.log
2024-05-08T01:37:56.649 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% 80.2% -- replaced with /var/log/ceph/cluster1-client.admin.63432.log.gz
2024-05-08T01:37:56.649 INFO:teuthology.orchestra.run.smithi142.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68522.log.gz
2024-05-08T01:37:56.649 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.48890.log.gz
2024-05-08T01:37:56.649 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.32267.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65717.log
2024-05-08T01:37:56.649 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.32267.log.gz
2024-05-08T01:37:56.649 INFO:teuthology.orchestra.run.smithi142.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.58240.log.gz
2024-05-08T01:37:56.650 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.44255.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35725.log
2024-05-08T01:37:56.650 INFO:teuthology.orchestra.run.smithi142.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.admin.44255.log.gz
2024-05-08T01:37:56.650 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.65717.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31260.log
2024-05-08T01:37:56.650 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.30280.log
2024-05-08T01:37:56.650 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.35725.log:	/var/log/ceph/cluster2-client.mirror.31260.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42694.log
2024-05-08T01:37:56.650 INFO:teuthology.orchestra.run.smithi142.stderr: 89.3% 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.31260.log.gz
2024-05-08T01:37:56.650 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.65717.log.gz 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.35725.log.gz
2024-05-08T01:37:56.650 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:56.650 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.1.30280.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59296.log
2024-05-08T01:37:56.650 INFO:teuthology.orchestra.run.smithi142.stderr: 20.5% -- replaced with /var/log/ceph/cluster1-client.mirror.1.30280.log.gz
2024-05-08T01:37:56.650 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.42694.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70586.log
2024-05-08T01:37:56.650 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.59296.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44854.log
2024-05-08T01:37:56.650 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.42694.log.gz
2024-05-08T01:37:56.651 INFO:teuthology.orchestra.run.smithi142.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35809.log.gz
2024-05-08T01:37:56.651 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.70586.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37117.log
2024-05-08T01:37:56.651 INFO:teuthology.orchestra.run.smithi142.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59296.log.gz
2024-05-08T01:37:56.651 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70586.log.gz
2024-05-08T01:37:56.651 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.44854.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58731.log
2024-05-08T01:37:56.651 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44854.log.gz
2024-05-08T01:37:56.651 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46163.log
2024-05-08T01:37:56.651 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.37117.log:	 94.5% -- replaced with /var/log/ceph/cluster2-client.mirror.3.35334.log.gz
2024-05-08T01:37:56.651 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58731.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52881.log
2024-05-08T01:37:56.651 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.37117.log.gz
2024-05-08T01:37:56.651 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63550.log
2024-05-08T01:37:56.651 INFO:teuthology.orchestra.run.smithi142.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58731.log.gz
2024-05-08T01:37:56.651 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.46163.log:	/var/log/ceph/cluster1-client.mirror.52881.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60168.log
2024-05-08T01:37:56.651 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49197.log
2024-05-08T01:37:56.651 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.63550.log:	 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52881.log.gz
2024-05-08T01:37:56.652 INFO:teuthology.orchestra.run.smithi142.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.admin.46163.log.gz
2024-05-08T01:37:56.652 INFO:teuthology.orchestra.run.smithi142.stderr: 53.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63550.log.gz
2024-05-08T01:37:56.652 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.60168.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.73656.log
2024-05-08T01:37:56.652 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.49197.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62594.log
2024-05-08T01:37:56.652 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.3.73656.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54614.log
2024-05-08T01:37:56.652 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.49197.log.gz
2024-05-08T01:37:56.652 INFO:teuthology.orchestra.run.smithi142.stderr: 19.9% -- replaced with /var/log/ceph/cluster2-client.mirror.3.73656.log.gz
2024-05-08T01:37:56.652 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.62594.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67332.log
2024-05-08T01:37:56.652 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.54614.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62316.log
2024-05-08T01:37:56.652 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.54614.log.gz
2024-05-08T01:37:56.652 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62594.log.gz
2024-05-08T01:37:56.652 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55644.log
2024-05-08T01:37:56.652 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.67332.log:	/var/log/ceph/cluster2-client.mirror.62316.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59486.log
2024-05-08T01:37:56.652 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.67332.log.gz
2024-05-08T01:37:56.653 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.55644.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31313.log
2024-05-08T01:37:56.653 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60809.log
2024-05-08T01:37:56.653 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.59486.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.55644.log.gz
2024-05-08T01:37:56.653 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.31313.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58894.log
2024-05-08T01:37:56.653 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59486.log.gz
2024-05-08T01:37:56.653 INFO:teuthology.orchestra.run.smithi142.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62316.log.gz
2024-05-08T01:37:56.653 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46015.log
2024-05-08T01:37:56.653 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.60809.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68058.log
2024-05-08T01:37:56.653 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58894.log:	/var/log/ceph/cluster1-client.admin.46015.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.35202.log
2024-05-08T01:37:56.653 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.68058.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60236.log
2024-05-08T01:37:56.653 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.46015.log.gz
2024-05-08T01:37:56.653 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.0.35202.log:	 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.60809.log.gz
2024-05-08T01:37:56.653 INFO:teuthology.orchestra.run.smithi142.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58894.log.gz
2024-05-08T01:37:56.654 INFO:teuthology.orchestra.run.smithi142.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68058.log.gz
2024-05-08T01:37:56.654 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32046.log
2024-05-08T01:37:56.654 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.60236.log:	gzip 94.0% -5 -- replaced with /var/log/ceph/cluster2-client.mirror.60168.log.gz
2024-05-08T01:37:56.654 INFO:teuthology.orchestra.run.smithi142.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.48656.log
2024-05-08T01:37:56.654 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60236.log.gz
2024-05-08T01:37:56.654 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.32046.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56790.log
2024-05-08T01:37:56.654 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.32046.log.gz
2024-05-08T01:37:56.654 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.48656.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30092.log
2024-05-08T01:37:56.654 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.56790.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36971.log
2024-05-08T01:37:56.654 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.48656.log.gz
2024-05-08T01:37:56.654 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.30092.log:	 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56790.log.gz
2024-05-08T01:37:56.654 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32662.log
2024-05-08T01:37:56.654 INFO:teuthology.orchestra.run.smithi142.stderr: 80.0% -- replaced with /var/log/ceph/cluster2-client.admin.30092.log.gz
2024-05-08T01:37:56.654 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.36971.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41623.log
2024-05-08T01:37:56.655 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32174.log
2024-05-08T01:37:56.655 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.32662.log:	 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36971.log.gz
2024-05-08T01:37:56.655 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.41623.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35005.log
2024-05-08T01:37:56.655 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40494.log
2024-05-08T01:37:56.655 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.41623.log.gz
2024-05-08T01:37:56.655 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.32174.log:	 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.32662.log.gz/var/log/ceph/cluster2-client.mirror.35005.log:
2024-05-08T01:37:56.655 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47372.log
2024-05-08T01:37:56.655 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.32174.log.gz
2024-05-08T01:37:56.655 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.35005.log.gz
2024-05-08T01:37:56.655 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56950.log
2024-05-08T01:37:56.655 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.40494.log:	/var/log/ceph/cluster1-client.mirror.47372.log:	 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40494.log.gz
2024-05-08T01:37:56.655 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56032.log
2024-05-08T01:37:56.655 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.admin.56950.log:	 -- /var/log/ceph/cluster2-client.admin.29025.log
2024-05-08T01:37:56.655 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.56032.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71941.log
2024-05-08T01:37:56.656 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47372.log.gz
2024-05-08T01:37:56.656 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.56950.log.gz
2024-05-08T01:37:56.656 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.29025.log:	gzip 87.8%  0.0% -5 --verbose -- replaced with /var/log/ceph/cluster2-client.admin.29025.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.56032.log.gz --
2024-05-08T01:37:56.656 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:56.656 INFO:teuthology.orchestra.run.smithi142.stderr: /var/log/ceph/cluster2-client.mirror.43824.log
2024-05-08T01:37:56.656 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.71941.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71973.log
2024-05-08T01:37:56.656 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.43824.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33494.log
2024-05-08T01:37:56.656 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43824.log.gz
2024-05-08T01:37:56.656 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.71973.log:	 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71941.log.gz
2024-05-08T01:37:56.656 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58926.log
2024-05-08T01:37:56.656 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.33494.log:	 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71973.log.gz
2024-05-08T01:37:56.656 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34129.log
2024-05-08T01:37:56.656 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.58926.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65739.log
2024-05-08T01:37:56.657 INFO:teuthology.orchestra.run.smithi142.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.admin.33494.log.gz
2024-05-08T01:37:56.657 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.34129.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72916.log
2024-05-08T01:37:56.657 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.65739.log:	 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34129.log.gz
2024-05-08T01:37:56.657 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67106.log
2024-05-08T01:37:56.657 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58926.log.gz
2024-05-08T01:37:56.657 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.72916.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58874.log
2024-05-08T01:37:56.657 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65739.log.gz
2024-05-08T01:37:56.657 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.67106.log:	 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72916.log.gz
2024-05-08T01:37:56.657 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.66675.log
2024-05-08T01:37:56.657 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58874.log:	 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.67106.log.gz
2024-05-08T01:37:56.657 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69323.log
2024-05-08T01:37:56.657 INFO:teuthology.orchestra.run.smithi142.stderr: 94.4%/var/log/ceph/cluster1-client.mirror.0.66675.log:	 -- replaced with /var/log/ceph/cluster1-client.mirror.3.31439.log.gz
2024-05-08T01:37:56.657 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43984.log
2024-05-08T01:37:56.657 INFO:teuthology.orchestra.run.smithi142.stderr: 19.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.66675.log.gz
2024-05-08T01:37:56.658 INFO:teuthology.orchestra.run.smithi142.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58874.log.gz
2024-05-08T01:37:56.658 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.69323.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42196.log
2024-05-08T01:37:56.658 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.43984.log:	 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69323.log.gz
2024-05-08T01:37:56.658 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49811.log
2024-05-08T01:37:56.658 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- 88.5% /var/log/ceph/cluster1-client.mirror.52936.log -- replaced with /var/log/ceph/cluster2-client.admin.43984.log.gz
2024-05-08T01:37:56.658 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:56.658 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.42196.log:	/var/log/ceph/cluster2-client.admin.49811.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60147.log
2024-05-08T01:37:56.658 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.42196.log.gz
2024-05-08T01:37:56.658 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.52936.log:	 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.49811.log.gz
2024-05-08T01:37:56.658 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37458.log
2024-05-08T01:37:56.658 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52936.log.gz
2024-05-08T01:37:56.658 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.60147.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36675.log
2024-05-08T01:37:56.658 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.37458.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49343.log
2024-05-08T01:37:56.659 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.37458.log.gz
2024-05-08T01:37:56.659 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.36675.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43472.log
2024-05-08T01:37:56.659 INFO:teuthology.orchestra.run.smithi142.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60147.log.gz
2024-05-08T01:37:56.659 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73607.log
2024-05-08T01:37:56.659 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.49343.log:	 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36675.log.gz
2024-05-08T01:37:56.659 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.43472.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61882.log
2024-05-08T01:37:56.659 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.49343.log.gz
2024-05-08T01:37:56.659 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.73607.log:	 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.43472.log.gz
2024-05-08T01:37:56.659 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33463.log
2024-05-08T01:37:56.659 INFO:teuthology.orchestra.run.smithi142.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73607.log.gz
2024-05-08T01:37:56.659 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63626.log
2024-05-08T01:37:56.659 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.61882.log:	/var/log/ceph/cluster1-client.admin.33463.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43634.log
2024-05-08T01:37:56.659 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.63626.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.33463.log.gz
2024-05-08T01:37:56.660 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48233.log
2024-05-08T01:37:56.660 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.43634.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74174.log
2024-05-08T01:37:56.660 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.48233.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42789.log
2024-05-08T01:37:56.660 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.43634.log.gz
2024-05-08T01:37:56.660 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.74174.log:	 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.48233.log.gz
2024-05-08T01:37:56.660 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40015.log
2024-05-08T01:37:56.660 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.74174.log.gz
2024-05-08T01:37:56.660 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.42789.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42860.log
2024-05-08T01:37:56.660 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42789.log.gz
2024-05-08T01:37:56.660 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33377.log
2024-05-08T01:37:56.660 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.42860.log:	 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.42860.log.gz
2024-05-08T01:37:56.660 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.40015.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.30308.log
2024-05-08T01:37:56.660 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40015.log.gz
2024-05-08T01:37:56.660 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.33377.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52648.log
2024-05-08T01:37:56.661 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.33377.log.gz
2024-05-08T01:37:56.661 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.2.30308.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39563.log
2024-05-08T01:37:56.661 INFO:teuthology.orchestra.run.smithi142.stderr: 20.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.30308.log.gz
2024-05-08T01:37:56.661 INFO:teuthology.orchestra.run.smithi142.stderr:gzip/var/log/ceph/cluster1-client.admin.52648.log:	 -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61258.log
2024-05-08T01:37:56.661 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.39563.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57420.log
2024-05-08T01:37:56.661 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.52648.log.gz
2024-05-08T01:37:56.661 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39563.log.gz/var/log/ceph/cluster2-client.mirror.61258.log:
2024-05-08T01:37:56.661 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62065.log
2024-05-08T01:37:56.661 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.57420.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36231.log
2024-05-08T01:37:56.661 INFO:teuthology.orchestra.run.smithi142.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57420.log.gz/var/log/ceph/cluster2-client.mirror.62065.log:
2024-05-08T01:37:56.661 INFO:teuthology.orchestra.run.smithi142.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61258.log.gz
2024-05-08T01:37:56.661 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68078.log
2024-05-08T01:37:56.661 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41530.log
2024-05-08T01:37:56.661 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.36231.log:	 89.9%/var/log/ceph/cluster1-client.mirror.68078.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.62065.log.gzgzip
2024-05-08T01:37:56.662 INFO:teuthology.orchestra.run.smithi142.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.40141.log
2024-05-08T01:37:56.662 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.41530.log:	 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.41530.log.gz
2024-05-08T01:37:56.699 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46601.log
2024-05-08T01:37:56.699 INFO:teuthology.orchestra.run.smithi142.stderr: 92.0% 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.61882.log.gz
2024-05-08T01:37:56.699 INFO:teuthology.orchestra.run.smithi142.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63626.log.gz
2024-05-08T01:37:56.699 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.40141.log:	 -- replaced with /var/log/ceph/cluster1-client.mirror.36231.log.gz
2024-05-08T01:37:56.699 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42663.log
2024-05-08T01:37:56.699 INFO:teuthology.orchestra.run.smithi142.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68078.log.gz
2024-05-08T01:37:56.699 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55150.log
2024-05-08T01:37:56.700 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0%/var/log/ceph/cluster2-client.admin.46601.log:	 -- replaced with /var/log/ceph/cluster1-client.admin.40141.log.gz
2024-05-08T01:37:56.700 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.42663.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46919.log
2024-05-08T01:37:56.700 INFO:teuthology.orchestra.run.smithi142.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.admin.46601.log.gz
2024-05-08T01:37:56.700 INFO:teuthology.orchestra.run.smithi142.stderr:gzip/var/log/ceph/cluster1-client.admin.55150.log:	 -5 --verbose -- /var/log/ceph/cluster1-client.admin.73173.log
2024-05-08T01:37:56.700 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.42663.log.gz
2024-05-08T01:37:56.700 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.46919.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39692.log
2024-05-08T01:37:56.700 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.55150.log.gz
2024-05-08T01:37:56.700 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43859.log
2024-05-08T01:37:56.700 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.73173.log:	/var/log/ceph/cluster2-client.admin.39692.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33560.log
2024-05-08T01:37:56.700 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46919.log.gz
2024-05-08T01:37:56.700 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.73173.log.gz
2024-05-08T01:37:56.700 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.43859.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44226.log
2024-05-08T01:37:56.700 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.39692.log.gz
2024-05-08T01:37:56.701 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62296.log
2024-05-08T01:37:56.701 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43859.log.gz
2024-05-08T01:37:56.701 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.33560.log:	/var/log/ceph/cluster1-client.admin.44226.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53699.log
2024-05-08T01:37:56.701 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.33560.log.gz
2024-05-08T01:37:56.701 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.62296.log:	 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.44226.log.gz
2024-05-08T01:37:56.701 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31341.log
2024-05-08T01:37:56.701 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40399.log
2024-05-08T01:37:56.701 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.53699.log:	 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62296.log.gz
2024-05-08T01:37:56.701 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.31341.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36556.log
2024-05-08T01:37:56.701 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.53699.log.gz
2024-05-08T01:37:56.701 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.31341.log.gz
2024-05-08T01:37:56.701 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54260.log
2024-05-08T01:37:56.701 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.40399.log:	/var/log/ceph/cluster1-client.admin.36556.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38708.log
2024-05-08T01:37:56.701 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.40399.log.gz
2024-05-08T01:37:56.702 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.54260.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62983.log
2024-05-08T01:37:56.702 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.36556.log.gz
2024-05-08T01:37:56.702 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.54260.log.gz
2024-05-08T01:37:56.702 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.38708.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30143.log
2024-05-08T01:37:56.702 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66756.log
2024-05-08T01:37:56.702 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.38708.log.gz
2024-05-08T01:37:56.702 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.62983.log:	/var/log/ceph/cluster2-client.mirror.30143.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48426.log
2024-05-08T01:37:56.702 INFO:teuthology.orchestra.run.smithi142.stderr: 55.3% -- replaced with /var/log/ceph/cluster2-client.mirror.30143.log.gz
2024-05-08T01:37:56.702 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.66756.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58031.log
2024-05-08T01:37:56.702 INFO:teuthology.orchestra.run.smithi142.stderr: 82.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66756.log.gz
2024-05-08T01:37:56.702 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.48426.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61466.log
2024-05-08T01:37:56.702 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.58031.log:	 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.48426.log.gz
2024-05-08T01:37:56.702 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57929.log
2024-05-08T01:37:56.703 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log
2024-05-08T01:37:56.703 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.61466.log:	 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62983.log.gz
2024-05-08T01:37:56.703 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.57929.log:	 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61466.log.gz
2024-05-08T01:37:56.703 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43204.log
2024-05-08T01:37:56.703 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-osd.1.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31792.log
2024-05-08T01:37:56.703 INFO:teuthology.orchestra.run.smithi142.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.admin.57929.log.gz
2024-05-08T01:37:56.703 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.43204.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63325.log
2024-05-08T01:37:56.703 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.31792.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40047.log
2024-05-08T01:37:56.703 INFO:teuthology.orchestra.run.smithi142.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43204.log.gz
2024-05-08T01:37:56.703 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.31792.log.gz
2024-05-08T01:37:56.703 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.63325.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48292.log
2024-05-08T01:37:56.703 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63325.log.gz
2024-05-08T01:37:56.703 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.40047.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67133.log
2024-05-08T01:37:56.703 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.48292.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72968.log
2024-05-08T01:37:56.704 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.40047.log.gz
2024-05-08T01:37:56.704 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.67133.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55006.log
2024-05-08T01:37:56.704 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3%/var/log/ceph/cluster2-client.mirror.72968.log:	 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67133.log.gz
2024-05-08T01:37:56.704 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49842.log
2024-05-08T01:37:56.704 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.48292.log.gz
2024-05-08T01:37:56.704 INFO:teuthology.orchestra.run.smithi142.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72968.log.gz
2024-05-08T01:37:56.704 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.55006.log:	 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.55006.log.gz
2024-05-08T01:37:56.704 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60480.log
2024-05-08T01:37:56.704 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.49842.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46222.log
2024-05-08T01:37:56.704 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.60480.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38570.log
2024-05-08T01:37:56.704 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.49842.log.gz
2024-05-08T01:37:56.704 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39776.log
2024-05-08T01:37:56.704 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.46222.log:	 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60480.log.gz
2024-05-08T01:37:56.704 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56452.log
2024-05-08T01:37:56.705 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.38570.log:	/var/log/ceph/cluster1-client.admin.39776.log:	 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46222.log.gz
2024-05-08T01:37:56.705 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54788.log
2024-05-08T01:37:56.705 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.56452.log:	 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.38570.log.gz
2024-05-08T01:37:56.705 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.39776.log.gz
2024-05-08T01:37:56.705 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.56452.log.gz
2024-05-08T01:37:56.705 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62641.log
2024-05-08T01:37:56.705 INFO:teuthology.orchestra.run.smithi142.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58031.log.gz
2024-05-08T01:37:56.705 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56060.log
2024-05-08T01:37:56.705 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.54788.log:	/var/log/ceph/cluster1-client.mirror.62641.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40743.log
2024-05-08T01:37:56.705 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.54788.log.gz
2024-05-08T01:37:56.705 INFO:teuthology.orchestra.run.smithi142.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62641.log.gz
2024-05-08T01:37:56.705 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.56060.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72143.log
2024-05-08T01:37:56.705 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38263.log
2024-05-08T01:37:56.705 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.40743.log:	 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56060.log.gz
2024-05-08T01:37:56.705 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.72143.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61142.log
2024-05-08T01:37:56.706 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.40743.log.gz
2024-05-08T01:37:56.706 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56147.log
2024-05-08T01:37:56.706 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72143.log.gz
2024-05-08T01:37:56.706 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.38263.log:	/var/log/ceph/cluster1-client.mirror.61142.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38171.log
2024-05-08T01:37:56.706 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.38263.log.gz
2024-05-08T01:37:56.706 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41139.log
2024-05-08T01:37:56.706 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61142.log.gz
2024-05-08T01:37:56.706 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.56147.log:	/var/log/ceph/cluster2-client.mirror.38171.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52147.log
2024-05-08T01:37:56.706 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.56147.log.gz
2024-05-08T01:37:56.706 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38171.log.gz
2024-05-08T01:37:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.41139.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40529.log
2024-05-08T01:37:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.52147.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47313.log
2024-05-08T01:37:56.707 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.41139.log.gz
2024-05-08T01:37:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.40529.log:	 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.52147.log.gz
2024-05-08T01:37:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50874.log
2024-05-08T01:37:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73317.log
2024-05-08T01:37:56.707 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4%/var/log/ceph/cluster2-client.admin.47313.log:	 -- replaced with /var/log/ceph/cluster1-client.mirror.40529.log.gz
2024-05-08T01:37:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.50874.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55094.log
2024-05-08T01:37:56.707 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.47313.log.gz
2024-05-08T01:37:56.707 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.50874.log.gz
2024-05-08T01:37:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.73317.log:	 /var/log/ceph/cluster1-client.mirror.61731.log
2024-05-08T01:37:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.55094.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61819.log
2024-05-08T01:37:56.707 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73317.log.gz
2024-05-08T01:37:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.61731.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51052.log
2024-05-08T01:37:56.707 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55094.log.gz
2024-05-08T01:37:56.708 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.61819.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33035.log
2024-05-08T01:37:56.708 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.61731.log.gz
2024-05-08T01:37:56.708 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66236.log
2024-05-08T01:37:56.708 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.51052.log:	/var/log/ceph/cluster1-client.admin.33035.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53305.log
2024-05-08T01:37:56.708 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51052.log.gz
2024-05-08T01:37:56.708 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61819.log.gz
2024-05-08T01:37:56.708 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.66236.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.33035.log.gz
2024-05-08T01:37:56.708 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68683.log
2024-05-08T01:37:56.708 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.53305.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66236.log.gz
2024-05-08T01:37:56.708 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39239.log
2024-05-08T01:37:56.708 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.68683.log:	 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.53305.log.gz
2024-05-08T01:37:56.708 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70301.log
2024-05-08T01:37:56.708 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68683.log.gz
2024-05-08T01:37:56.708 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40892.log
2024-05-08T01:37:56.709 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.39239.log:	/var/log/ceph/cluster1-client.mirror.70301.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46891.log
2024-05-08T01:37:56.709 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.39239.log.gz
2024-05-08T01:37:56.709 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.40892.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67637.log
2024-05-08T01:37:56.709 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.46891.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44752.log
2024-05-08T01:37:56.709 INFO:teuthology.orchestra.run.smithi142.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40892.log.gz
2024-05-08T01:37:56.709 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52971.log/var/log/ceph/cluster2-client.mirror.67637.log:
2024-05-08T01:37:56.709 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.46891.log.gz/var/log/ceph/cluster2-client.mirror.44752.log:	 87.4%
2024-05-08T01:37:56.709 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54819.log
2024-05-08T01:37:56.709 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.67637.log.gz
2024-05-08T01:37:56.709 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log
2024-05-08T01:37:56.709 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.52971.log:	 91.4% 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44752.log.gz
2024-05-08T01:37:56.709 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70301.log.gz
2024-05-08T01:37:56.709 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.54819.log:	 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52971.log.gz
2024-05-08T01:37:56.709 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48332.log
2024-05-08T01:37:56.710 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-mon.a.log:	 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.54819.log.gz
2024-05-08T01:37:56.710 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47036.log
2024-05-08T01:37:56.710 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.48332.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35753.log
2024-05-08T01:37:56.710 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.47036.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67932.log
2024-05-08T01:37:56.710 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.48332.log.gz
2024-05-08T01:37:56.710 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.35753.log:	 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.47036.log.gz
2024-05-08T01:37:56.710 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44952.log
2024-05-08T01:37:56.710 INFO:teuthology.orchestra.run.smithi142.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35753.log.gz
2024-05-08T01:37:56.710 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45097.log
2024-05-08T01:37:56.710 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.67932.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.30275.log
2024-05-08T01:37:56.710 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.44952.log:	/var/log/ceph/cluster1-client.admin.45097.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39720.log
2024-05-08T01:37:56.710 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.67932.log.gz
2024-05-08T01:37:56.710 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.44952.log.gz
2024-05-08T01:37:56.710 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.0.30275.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.45097.log.gz
2024-05-08T01:37:56.711 INFO:teuthology.orchestra.run.smithi142.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.30275.log.gz
2024-05-08T01:37:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61572.log
2024-05-08T01:37:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.39720.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38949.log
2024-05-08T01:37:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.61572.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71514.log
2024-05-08T01:37:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.38949.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43532.log
2024-05-08T01:37:56.711 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39720.log.gz
2024-05-08T01:37:56.711 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61572.log.gz
2024-05-08T01:37:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.71514.log:	 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.38949.log.gz
2024-05-08T01:37:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38542.log
2024-05-08T01:37:56.711 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71514.log.gz
2024-05-08T01:37:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.43532.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64386.log
2024-05-08T01:37:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.38542.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66625.log
2024-05-08T01:37:56.711 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.43532.log.gz
2024-05-08T01:37:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.64386.log:	 87.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72545.log
2024-05-08T01:37:56.712 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.38542.log.gz
2024-05-08T01:37:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.66625.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60096.log
2024-05-08T01:37:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.72545.log:	 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66625.log.gz
2024-05-08T01:37:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69224.log
2024-05-08T01:37:56.712 INFO:teuthology.orchestra.run.smithi142.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.64386.log.gz
2024-05-08T01:37:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:gzip/var/log/ceph/cluster2-client.mirror.60096.log:	 -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30765.log
2024-05-08T01:37:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.69224.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64218.log
2024-05-08T01:37:56.712 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60096.log.gz
2024-05-08T01:37:56.712 INFO:teuthology.orchestra.run.smithi142.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72545.log.gz
2024-05-08T01:37:56.712 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.69224.log.gz
2024-05-08T01:37:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.30765.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58763.log
2024-05-08T01:37:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.64218.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46099.log
2024-05-08T01:37:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.58763.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72389.log
2024-05-08T01:37:56.713 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.64218.log.gz
2024-05-08T01:37:56.713 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46831.log
2024-05-08T01:37:56.713 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.46099.log:	/var/log/ceph/cluster1-client.mirror.72389.log:	 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.58763.log.gz
2024-05-08T01:37:56.713 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.69643.log
2024-05-08T01:37:56.713 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.46099.log.gz
2024-05-08T01:37:56.713 INFO:teuthology.orchestra.run.smithi142.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.72389.log.gz
2024-05-08T01:37:56.713 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.46831.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55515.log
2024-05-08T01:37:56.713 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.0.69643.log:	 88.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40774.log
2024-05-08T01:37:56.713 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.46831.log.gz
2024-05-08T01:37:56.713 INFO:teuthology.orchestra.run.smithi142.stderr: 20.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.69643.log.gz
2024-05-08T01:37:56.713 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.55515.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50695.log
2024-05-08T01:37:56.713 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.40774.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.55515.log.gz
2024-05-08T01:37:56.713 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31106.log
2024-05-08T01:37:56.713 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51182.log
2024-05-08T01:37:56.714 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.40774.log.gz
2024-05-08T01:37:56.714 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.50695.log:	/var/log/ceph/cluster1-client.mirror.31106.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54450.log
2024-05-08T01:37:56.714 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.51182.log:	 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.50695.log.gz
2024-05-08T01:37:56.714 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29478.log
2024-05-08T01:37:56.714 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.51182.log.gz
2024-05-08T01:37:56.714 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.54450.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47549.log
2024-05-08T01:37:56.714 INFO:teuthology.orchestra.run.smithi142.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.31106.log.gz
2024-05-08T01:37:56.714 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.29478.log:	  0.0% 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54450.log.gz
2024-05-08T01:37:56.714 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.29478.log.gz
2024-05-08T01:37:56.714 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61040.log
2024-05-08T01:37:56.714 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.47549.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52676.log
2024-05-08T01:37:56.714 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59987.log
2024-05-08T01:37:56.714 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.47549.log.gz
2024-05-08T01:37:56.715 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.61040.log:	/var/log/ceph/cluster2-client.mirror.52676.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69002.log
2024-05-08T01:37:56.715 INFO:teuthology.orchestra.run.smithi142.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30765.log.gz
2024-05-08T01:37:56.715 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.59987.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57358.log
2024-05-08T01:37:56.715 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61040.log.gz
2024-05-08T01:37:56.715 INFO:teuthology.orchestra.run.smithi142.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52676.log.gz
2024-05-08T01:37:56.715 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.59987.log.gz
2024-05-08T01:37:56.715 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.69002.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56237.log
2024-05-08T01:37:56.715 INFO:teuthology.orchestra.run.smithi142.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69002.log.gz
2024-05-08T01:37:56.715 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.57358.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50297.log
2024-05-08T01:37:56.715 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.56237.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35202.log
2024-05-08T01:37:56.715 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.57358.log.gz
2024-05-08T01:37:56.715 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.50297.log:	 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.56237.log.gz
2024-05-08T01:37:56.715 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70397.log
2024-05-08T01:37:56.715 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.35202.log:	 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.50297.log.gz
2024-05-08T01:37:56.716 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44165.log
2024-05-08T01:37:56.716 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.70397.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42824.log
2024-05-08T01:37:56.716 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cluster1-osd.1.log:	 91.1% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz
2024-05-08T01:37:56.743 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42284.log
2024-05-08T01:37:56.743 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.44165.log:	/var/log/ceph/cluster2-client.mirror.42824.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.44165.log.gz
2024-05-08T01:37:56.743 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42824.log.gz
2024-05-08T01:37:56.743 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72472.log
2024-05-08T01:37:56.743 INFO:teuthology.orchestra.run.smithi142.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70397.log.gz
2024-05-08T01:37:56.743 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42256.log
2024-05-08T01:37:56.743 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.42284.log:	/var/log/ceph/cluster2-client.mirror.72472.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49556.log
2024-05-08T01:37:56.744 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.42256.log:	 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72472.log.gz
2024-05-08T01:37:56.744 INFO:teuthology.orchestra.run.smithi142.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42284.log.gz
2024-05-08T01:37:56.744 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.42256.log.gz
2024-05-08T01:37:56.744 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39075.log
2024-05-08T01:37:56.744 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72778.log
2024-05-08T01:37:56.744 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.49556.log:	/var/log/ceph/cluster2-client.mirror.39075.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73042.log
2024-05-08T01:37:56.744 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.49556.log.gz
2024-05-08T01:37:56.744 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39075.log.gz
2024-05-08T01:37:56.744 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.72778.log:	 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72778.log.gz
2024-05-08T01:37:56.744 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63083.log
2024-05-08T01:37:56.744 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.73042.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37997.log
2024-05-08T01:37:56.744 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.63083.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54485.log
2024-05-08T01:37:56.745 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.73042.log.gz
2024-05-08T01:37:56.745 INFO:teuthology.orchestra.run.smithi142.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63083.log.gz
2024-05-08T01:37:56.745 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.37997.log:	 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37997.log.gz
2024-05-08T01:37:56.745 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42953.log
2024-05-08T01:37:56.745 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.54485.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46393.log
2024-05-08T01:37:56.745 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54485.log.gz/var/log/ceph/cluster1-client.admin.42953.log:
2024-05-08T01:37:56.745 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54047.log
2024-05-08T01:37:56.745 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.46393.log:	 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.42953.log.gz
2024-05-08T01:37:56.745 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.46393.log.gz
2024-05-08T01:37:56.745 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32139.log
2024-05-08T01:37:56.745 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.54047.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38136.log
2024-05-08T01:37:56.745 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54047.log.gz/var/log/ceph/cluster2-client.mirror.32139.log:
2024-05-08T01:37:56.746 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41284.log
2024-05-08T01:37:56.746 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.32139.log.gz
2024-05-08T01:37:56.746 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.38136.log:	 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38136.log.gz
2024-05-08T01:37:56.746 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29955.log
2024-05-08T01:37:56.746 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.41284.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73342.log
2024-05-08T01:37:56.746 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.29955.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63138.log
2024-05-08T01:37:56.746 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.41284.log.gz
2024-05-08T01:37:56.746 INFO:teuthology.orchestra.run.smithi142.stderr:  4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.29955.log.gz
2024-05-08T01:37:56.746 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.73342.log:	 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73342.log.gz
2024-05-08T01:37:56.746 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59219.log
2024-05-08T01:37:56.746 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.63138.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35939.log
2024-05-08T01:37:56.746 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.59219.log:	 88.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30833.log
2024-05-08T01:37:56.747 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.63138.log.gz
2024-05-08T01:37:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.35939.log:	 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.35939.log.gz
2024-05-08T01:37:56.747 INFO:teuthology.orchestra.run.smithi142.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.59219.log.gz
2024-05-08T01:37:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39208.log
2024-05-08T01:37:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.30833.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32330.log
2024-05-08T01:37:56.747 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6%/var/log/ceph/cluster1-client.admin.39208.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.30833.log.gz
2024-05-08T01:37:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66822.log
2024-05-08T01:37:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.32330.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.39208.log.gz
2024-05-08T01:37:56.747 INFO:teuthology.orchestra.run.smithi142.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.admin.32330.log.gz
2024-05-08T01:37:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40803.log
2024-05-08T01:37:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.66822.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.69685.log
2024-05-08T01:37:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.40803.log:	 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.66822.log.gz
2024-05-08T01:37:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42630.log
2024-05-08T01:37:56.748 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29303.log
2024-05-08T01:37:56.748 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.40803.log.gz
2024-05-08T01:37:56.748 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.42630.log:	/var/log/ceph/cluster1-client.mirror.2.69685.log:	 19.9% -- replaced with /var/log/ceph/cluster1-client.mirror.2.69685.log.gz
2024-05-08T01:37:56.748 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62553.log
2024-05-08T01:37:56.748 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.42630.log.gz
2024-05-08T01:37:56.748 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.29303.log:	gzip  0.0% -5 -- replaced with /var/log/ceph/cluster2-client.admin.29303.log.gz --verbose
2024-05-08T01:37:56.748 INFO:teuthology.orchestra.run.smithi142.stderr: -- /var/log/ceph/cluster1-client.mirror.70752.log
2024-05-08T01:37:56.748 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.62553.log:	 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62553.log.gz
2024-05-08T01:37:56.748 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55614.log
2024-05-08T01:37:56.748 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.70752.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68298.log
2024-05-08T01:37:56.748 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.55614.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46975.log
2024-05-08T01:37:56.749 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.70752.log.gz
2024-05-08T01:37:56.749 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.68298.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.55614.log.gz
2024-05-08T01:37:56.749 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68298.log.gz
2024-05-08T01:37:56.749 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53970.log
2024-05-08T01:37:56.749 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.46975.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52082.log
2024-05-08T01:37:56.749 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.53970.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44890.log
2024-05-08T01:37:56.749 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.46975.log.gz
2024-05-08T01:37:56.749 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.52082.log:	 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53970.log.gz
2024-05-08T01:37:56.749 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52082.log.gz
2024-05-08T01:37:56.749 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59863.log
2024-05-08T01:37:56.749 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.44890.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32973.log
2024-05-08T01:37:56.750 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.59863.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61694.log
2024-05-08T01:37:56.750 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.44890.log.gz
2024-05-08T01:37:56.750 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.32973.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.32973.log.gz
2024-05-08T01:37:56.750 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40304.log
2024-05-08T01:37:56.750 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.61694.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58619.log
2024-05-08T01:37:56.750 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8%/var/log/ceph/cluster1-client.admin.40304.log:	 -- replaced with /var/log/ceph/cluster1-client.mirror.61694.log.gz 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59863.log.gz
2024-05-08T01:37:56.750 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:56.750 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.40304.log.gz
2024-05-08T01:37:56.750 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29870.log
2024-05-08T01:37:56.750 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58619.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29989.log
2024-05-08T01:37:56.750 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.29870.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53728.log
2024-05-08T01:37:56.750 INFO:teuthology.orchestra.run.smithi142.stderr: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.29870.log.gz
2024-05-08T01:37:56.751 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.29989.log:	 57.1% -- replaced with /var/log/ceph/cluster1-client.mirror.29989.log.gz
2024-05-08T01:37:56.751 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46860.log
2024-05-08T01:37:56.751 INFO:teuthology.orchestra.run.smithi142.stderr: 92.4%/var/log/ceph/cluster2-client.admin.53728.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.58619.log.gzgzip
2024-05-08T01:37:56.751 INFO:teuthology.orchestra.run.smithi142.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59843.log
2024-05-08T01:37:56.751 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.53728.log.gz
2024-05-08T01:37:56.751 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.46860.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.46860.log.gz
2024-05-08T01:37:56.751 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42390.log
2024-05-08T01:37:56.751 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.59843.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32081.log
2024-05-08T01:37:56.751 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.42390.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44385.log
2024-05-08T01:37:56.751 INFO:teuthology.orchestra.run.smithi142.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59843.log.gz
2024-05-08T01:37:56.751 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42390.log.gz
2024-05-08T01:37:56.752 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.32081.log:	 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.32081.log.gz
2024-05-08T01:37:56.752 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45408.log
2024-05-08T01:37:56.752 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.44385.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29428.log
2024-05-08T01:37:56.752 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.45408.log:	 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44385.log.gz
2024-05-08T01:37:56.752 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60201.log
2024-05-08T01:37:56.752 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36253.log
2024-05-08T01:37:56.752 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.29428.log:	 87.8%/var/log/ceph/cluster2-client.mirror.60201.log:	  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29428.log.gz
2024-05-08T01:37:56.752 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.45408.log.gz
2024-05-08T01:37:56.752 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60201.log.gz
2024-05-08T01:37:56.752 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46134.log
2024-05-08T01:37:56.752 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20425.log
2024-05-08T01:37:56.752 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.36253.log:	/var/log/ceph/cluster1-client.admin.46134.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46422.log
2024-05-08T01:37:56.752 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph.tmp-client.admin.20425.log:	  0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20425.log.gz
2024-05-08T01:37:56.753 INFO:teuthology.orchestra.run.smithi142.stderr: 90.1% 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.46134.log.gz
2024-05-08T01:37:56.753 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.36253.log.gz
2024-05-08T01:37:56.753 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.66684.log
2024-05-08T01:37:56.753 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.46422.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36297.log
2024-05-08T01:37:56.753 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.0.66684.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70215.log
2024-05-08T01:37:56.753 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.46422.log.gz
2024-05-08T01:37:56.753 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.36297.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.36297.log.gz
2024-05-08T01:37:56.753 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37810.log
2024-05-08T01:37:56.753 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.70215.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34743.log
2024-05-08T01:37:56.753 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.37810.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.70215.log.gz
2024-05-08T01:37:56.753 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.37810.log.gz
2024-05-08T01:37:56.753 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57548.log
2024-05-08T01:37:56.753 INFO:teuthology.orchestra.run.smithi142.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.35202.log.gz
2024-05-08T01:37:56.753 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.34743.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61776.log
2024-05-08T01:37:56.754 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.34743.log.gz
2024-05-08T01:37:56.754 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.57548.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61639.log
2024-05-08T01:37:56.754 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57548.log.gz/var/log/ceph/cluster2-client.mirror.61776.log:
2024-05-08T01:37:56.754 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52732.log
2024-05-08T01:37:56.754 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.61639.log:	 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61639.log.gz
2024-05-08T01:37:56.754 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62890.log
2024-05-08T01:37:56.754 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.52732.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37618.log
2024-05-08T01:37:56.754 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.62890.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.52732.log.gz
2024-05-08T01:37:56.754 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62890.log.gz
2024-05-08T01:37:56.754 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67823.log
2024-05-08T01:37:56.754 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.37618.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32724.log
2024-05-08T01:37:56.754 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.67823.log:	 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67823.log.gz
2024-05-08T01:37:56.754 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68801.log
2024-05-08T01:37:56.755 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.32724.log:	 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37618.log.gz
2024-05-08T01:37:56.755 INFO:teuthology.orchestra.run.smithi142.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.admin.32724.log.gz
2024-05-08T01:37:56.755 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64485.log
2024-05-08T01:37:56.755 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.68801.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72210.log
2024-05-08T01:37:56.755 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.64485.log:	 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64485.log.gz
2024-05-08T01:37:56.755 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35130.log
2024-05-08T01:37:56.768 INFO:teuthology.orchestra.run.smithi106.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz
2024-05-08T01:37:56.768 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.72210.log:	 92.3% 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.72210.log.gz
2024-05-08T01:37:56.768 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.68801.log.gz
2024-05-08T01:37:56.768 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.69652.log
2024-05-08T01:37:56.768 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.35130.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36496.log
2024-05-08T01:37:56.768 INFO:teuthology.orchestra.run.smithi142.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.61776.log.gz
2024-05-08T01:37:56.768 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.0.69652.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.35130.log.gz
2024-05-08T01:37:56.769 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67963.log
2024-05-08T01:37:56.769 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.36496.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43021.log
2024-05-08T01:37:56.769 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.67963.log:	 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36496.log.gz
2024-05-08T01:37:56.769 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.67963.log.gz
2024-05-08T01:37:56.769 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40240.log
2024-05-08T01:37:56.769 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71788.log
2024-05-08T01:37:56.769 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.43021.log:	/var/log/ceph/cluster1-client.admin.40240.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.43021.log.gz
2024-05-08T01:37:56.769 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.40240.log.gz
2024-05-08T01:37:56.769 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69053.log
2024-05-08T01:37:56.769 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.71788.log:	 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71788.log.gz
2024-05-08T01:37:56.769 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29653.log
2024-05-08T01:37:56.769 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74625.log
2024-05-08T01:37:56.769 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.69053.log:	/var/log/ceph/cluster2-client.admin.29653.log:	  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29653.log.gz
2024-05-08T01:37:56.770 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69053.log.gz
2024-05-08T01:37:56.770 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50724.log
2024-05-08T01:37:56.770 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74075.log
2024-05-08T01:37:56.770 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.74625.log:	/var/log/ceph/cluster1-client.admin.50724.log:	  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74625.log.gz
2024-05-08T01:37:56.770 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33982.log
2024-05-08T01:37:56.770 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.50724.log.gz
2024-05-08T01:37:56.770 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.74075.log:	  0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74075.log.gz
2024-05-08T01:37:56.770 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72293.log
2024-05-08T01:37:56.770 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42455.log
2024-05-08T01:37:56.770 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.33982.log:	/var/log/ceph/cluster1-client.mirror.72293.log:	 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72293.log.gz
2024-05-08T01:37:56.770 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.33982.log.gz
2024-05-08T01:37:56.770 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56320.log
2024-05-08T01:37:56.770 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49649.log
2024-05-08T01:37:56.771 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.42455.log:	/var/log/ceph/cluster2-client.mirror.56320.log:	 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.42455.log.gz
2024-05-08T01:37:56.771 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56320.log.gz
2024-05-08T01:37:56.771 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47910.log
2024-05-08T01:37:56.771 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34264.log
2024-05-08T01:37:56.771 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.49649.log:	/var/log/ceph/cluster1-client.admin.47910.log:	 88.0% 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.49649.log.gz
2024-05-08T01:37:56.771 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.47910.log.gz
2024-05-08T01:37:56.771 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44514.log
2024-05-08T01:37:56.771 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38676.log
2024-05-08T01:37:56.771 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.34264.log:	/var/log/ceph/cluster2-client.admin.44514.log:	 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.34264.log.gz
2024-05-08T01:37:56.771 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.44514.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51888.log
2024-05-08T01:37:56.771 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:56.771 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.38676.log:	 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38676.log.gz
2024-05-08T01:37:56.771 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68353.log
2024-05-08T01:37:56.772 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57708.log
2024-05-08T01:37:56.772 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.51888.log:	/var/log/ceph/cluster2-client.mirror.68353.log:	 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.51888.log.gz
2024-05-08T01:37:56.772 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68353.log.gz
2024-05-08T01:37:56.772 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58553.log
2024-05-08T01:37:56.772 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.66761.log
2024-05-08T01:37:56.772 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.57708.log:	/var/log/ceph/cluster1-client.admin.58553.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.57708.log.gz
2024-05-08T01:37:56.772 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.58553.log.gz
2024-05-08T01:37:56.772 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57066.log
2024-05-08T01:37:56.772 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.73651.log
2024-05-08T01:37:56.772 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.3.66761.log:	/var/log/ceph/cluster1-client.admin.57066.log:	 20.5% -- replaced with /var/log/ceph/cluster1-client.mirror.3.66761.log.gz
2024-05-08T01:37:56.772 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.57066.log.gz
2024-05-08T01:37:56.772 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59272.log
2024-05-08T01:37:56.772 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62691.log
2024-05-08T01:37:56.772 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.2.73651.log:	/var/log/ceph/cluster2-client.mirror.59272.log:	 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59272.log.gz
2024-05-08T01:37:56.777 INFO:teuthology.orchestra.run.smithi142.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.2.73651.log.gz
2024-05-08T01:37:56.777 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62048.log
2024-05-08T01:37:56.777 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43694.log
2024-05-08T01:37:56.777 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.62691.log:	/var/log/ceph/cluster2-client.mirror.62048.log:	 56.2% 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62691.log.gz
2024-05-08T01:37:56.777 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.62048.log.gz
2024-05-08T01:37:56.777 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35098.log
2024-05-08T01:37:56.777 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39385.log
2024-05-08T01:37:56.778 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.43694.log:	/var/log/ceph/cluster2-client.mirror.35098.log:	 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35098.log.gz 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.43694.log.gz
2024-05-08T01:37:56.778 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:56.778 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54116.log
2024-05-08T01:37:56.778 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54847.log
2024-05-08T01:37:56.778 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.39385.log:	/var/log/ceph/cluster1-client.admin.54116.log:	 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.39385.log.gz
2024-05-08T01:37:56.778 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.54116.log.gz
2024-05-08T01:37:56.778 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.31341.log
2024-05-08T01:37:56.778 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46194.log
2024-05-08T01:37:56.778 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.54847.log:	/var/log/ceph/cluster1-client.mirror.1.31341.log:	 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54847.log.gz
2024-05-08T01:37:56.778 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46360.log
2024-05-08T01:37:56.778 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.46194.log:	 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.46194.log.gz
2024-05-08T01:37:56.778 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45011.log
2024-05-08T01:37:56.778 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.46360.log:	 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.46360.log.gz
2024-05-08T01:37:56.779 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49169.log
2024-05-08T01:37:56.780 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.45011.log:	 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45011.log.gz
2024-05-08T01:37:56.780 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36797.log
2024-05-08T01:37:56.780 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.49169.log:	 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.49169.log.gz
2024-05-08T01:37:56.780 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52179.log
2024-05-08T01:37:56.780 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.36797.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.36797.log.gz
2024-05-08T01:37:56.781 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30891.log
2024-05-08T01:37:56.781 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.52179.log:	 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.52179.log.gz
2024-05-08T01:37:56.782 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70485.log
2024-05-08T01:37:56.782 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.30891.log:	 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.30891.log.gz
2024-05-08T01:37:56.783 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30303.log
2024-05-08T01:37:56.783 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.70485.log:	 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70485.log.gz
2024-05-08T01:37:56.784 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49109.log
2024-05-08T01:37:56.784 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.30303.log:	 86.5% -- replaced with /var/log/ceph/cluster2-client.mirror.30303.log.gz
2024-05-08T01:37:56.785 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59550.log
2024-05-08T01:37:56.785 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.49109.log:	 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.49109.log.gz
2024-05-08T01:37:56.786 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52389.log
2024-05-08T01:37:56.789 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.59550.log:	 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59550.log.gz
2024-05-08T01:37:56.789 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.31304.log
2024-05-08T01:37:56.790 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.52389.log:	 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.52389.log.gz
2024-05-08T01:37:56.790 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72712.log
2024-05-08T01:37:56.791 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.0.31304.log:	 19.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.31304.log.gz
2024-05-08T01:37:56.791 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57296.log
2024-05-08T01:37:56.792 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.72712.log:	 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72712.log.gz
2024-05-08T01:37:56.792 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43275.log
2024-05-08T01:37:56.793 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.57296.log:	 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.57296.log.gz
2024-05-08T01:37:56.794 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67900.log
2024-05-08T01:37:56.794 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.43275.log:	 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43275.log.gz
2024-05-08T01:37:56.795 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57329.log
2024-05-08T01:37:56.795 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.67900.log:	 86.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67900.log.gz
2024-05-08T01:37:56.796 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39807.log
2024-05-08T01:37:56.796 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.57329.log:	 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.57329.log.gz
2024-05-08T01:37:56.797 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31919.log
2024-05-08T01:37:56.798 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.39807.log:	 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.39807.log.gz
2024-05-08T01:37:56.798 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53040.log
2024-05-08T01:37:56.799 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.31919.log:	 90.6% -- replaced with /var/log/ceph/cluster2-client.admin.31919.log.gz
2024-05-08T01:37:56.800 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70246.log
2024-05-08T01:37:56.800 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.53040.log:	 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.53040.log.gz
2024-05-08T01:37:56.801 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48800.log
2024-05-08T01:37:56.802 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.70246.log:	 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.70246.log.gz
2024-05-08T01:37:56.802 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51644.log
2024-05-08T01:37:56.803 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.48800.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.48800.log.gz
2024-05-08T01:37:56.803 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73073.log
2024-05-08T01:37:56.804 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.51644.log:	 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.51644.log.gz
2024-05-08T01:37:56.805 INFO:teuthology.orchestra.run.smithi142.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.66684.log.gz
2024-05-08T01:37:56.805 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29887.log
2024-05-08T01:37:56.806 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.31318.log
2024-05-08T01:37:56.806 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.73073.log:	/var/log/ceph/cluster1-client.admin.29887.log:	 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.29887.log.gz
2024-05-08T01:37:56.806 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.73073.log.gz
2024-05-08T01:37:56.807 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51215.log
2024-05-08T01:37:56.808 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.1.31318.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.35274.log
2024-05-08T01:37:56.808 INFO:teuthology.orchestra.run.smithi142.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-client.mirror.1.31341.log.gz
2024-05-08T01:37:56.808 INFO:teuthology.orchestra.run.smithi142.stderr: 19.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.31318.log.gz
2024-05-08T01:37:56.808 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.51215.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30989.log
2024-05-08T01:37:56.808 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.2.35274.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55574.log
2024-05-08T01:37:56.808 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.51215.log.gz
2024-05-08T01:37:56.809 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.30989.log:	 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.69652.log.gz
2024-05-08T01:37:56.809 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.30989.log.gz
2024-05-08T01:37:56.809 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60329.log
2024-05-08T01:37:56.809 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.55574.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62196.log
2024-05-08T01:37:56.809 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.60329.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60071.log
2024-05-08T01:37:56.810 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60329.log.gz
2024-05-08T01:37:56.810 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55574.log.gz/var/log/ceph/cluster2-client.mirror.62196.log:
2024-05-08T01:37:56.811 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62196.log.gz
2024-05-08T01:37:56.811 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60997.log
2024-05-08T01:37:56.811 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.60071.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44923.log
2024-05-08T01:37:56.812 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.60997.log:	 87.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74831.log
2024-05-08T01:37:56.812 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.60071.log.gz
2024-05-08T01:37:56.812 INFO:teuthology.orchestra.run.smithi142.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60997.log.gz
2024-05-08T01:37:56.812 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.44923.log:	 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.44923.log.gz
2024-05-08T01:37:56.812 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41463.log
2024-05-08T01:37:56.816 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.74831.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50387.log
2024-05-08T01:37:56.816 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74831.log.gz
2024-05-08T01:37:56.816 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.41463.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44545.log
2024-05-08T01:37:56.816 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3%/var/log/ceph/cluster2-client.admin.50387.log:	 -- replaced with /var/log/ceph/cluster1-client.mirror.41463.log.gz
2024-05-08T01:37:56.817 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.73612.log
2024-05-08T01:37:56.817 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51738.log
2024-05-08T01:37:56.817 INFO:teuthology.orchestra.run.smithi142.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.0.35202.log.gz
2024-05-08T01:37:56.817 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.44545.log:	 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.50387.log.gz
2024-05-08T01:37:56.817 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.2.73612.log:	 20.5% -- replaced with /var/log/ceph/cluster2-client.mirror.2.73612.log.gz
2024-05-08T01:37:56.817 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.44545.log.gz
2024-05-08T01:37:56.817 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29251.log
2024-05-08T01:37:56.817 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.51738.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40335.log
2024-05-08T01:37:56.817 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.29251.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.51738.log.gz
2024-05-08T01:37:56.817 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.35193.log
2024-05-08T01:37:56.817 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29251.log.gz
2024-05-08T01:37:56.817 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.40335.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34382.log
2024-05-08T01:37:56.817 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.0.35193.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35661.log
2024-05-08T01:37:56.817 INFO:teuthology.orchestra.run.smithi142.stderr: 19.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.35193.log.gz
2024-05-08T01:37:56.818 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.40335.log.gz
2024-05-08T01:37:56.818 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.34382.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52294.log
2024-05-08T01:37:56.818 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.35661.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54902.log
2024-05-08T01:37:56.822 INFO:teuthology.orchestra.run.smithi142.stderr: 89.2%/var/log/ceph/cluster1-client.admin.52294.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.34382.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73733.log
2024-05-08T01:37:56.822 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7%
2024-05-08T01:37:56.823 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.35661.log.gz
2024-05-08T01:37:56.823 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.54902.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.52294.log.gz
2024-05-08T01:37:56.823 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54902.log.gz
2024-05-08T01:37:56.823 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72587.log
2024-05-08T01:37:56.823 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.73733.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69724.log
2024-05-08T01:37:56.823 INFO:teuthology.orchestra.run.smithi142.stderr: 27.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73733.log.gz
2024-05-08T01:37:56.823 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.72587.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62153.log
2024-05-08T01:37:56.823 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.69724.log:	gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log
2024-05-08T01:37:56.823 INFO:teuthology.orchestra.run.smithi142.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69724.log.gz
2024-05-08T01:37:56.823 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.62153.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53215.log
2024-05-08T01:37:56.823 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72587.log.gz
2024-05-08T01:37:56.823 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2.audit.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31439.log
2024-05-08T01:37:56.823 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.53215.log:	 89.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72654.log
2024-05-08T01:37:56.824 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.62153.log.gz
2024-05-08T01:37:56.824 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.31439.log:	 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.53215.log.gz
2024-05-08T01:37:56.824 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.31439.log.gz
2024-05-08T01:37:56.824 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31051.log
2024-05-08T01:37:56.824 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.72654.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36861.log
2024-05-08T01:37:56.824 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3%/var/log/ceph/cluster2-client.admin.31051.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.72654.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74651.log
2024-05-08T01:37:56.824 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:56.824 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.36861.log:	 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.31051.log.gz
2024-05-08T01:37:56.824 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.36861.log.gz
2024-05-08T01:37:56.824 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29024.log
2024-05-08T01:37:56.831 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37720.log
2024-05-08T01:37:56.831 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.74651.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41742.log
2024-05-08T01:37:56.831 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.29024.log:	  0.0%  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74651.log.gz
2024-05-08T01:37:56.831 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.29024.log.gz
2024-05-08T01:37:56.831 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.37720.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37423.log
2024-05-08T01:37:56.831 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66260.log
2024-05-08T01:37:56.831 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.37720.log.gz
2024-05-08T01:37:56.831 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.41742.log:	/var/log/ceph/cluster1-client.mirror.37423.log:	 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.41742.log.gz
2024-05-08T01:37:56.831 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37423.log.gz
2024-05-08T01:37:56.831 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log
2024-05-08T01:37:56.831 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54320.log
2024-05-08T01:37:56.832 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.66260.log:	/var/log/ceph/cluster2.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70935.log
2024-05-08T01:37:56.833 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.54320.log:	 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66260.log.gz
2024-05-08T01:37:56.833 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.54320.log.gz
2024-05-08T01:37:56.833 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29578.log
2024-05-08T01:37:56.833 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53069.log
2024-05-08T01:37:56.833 INFO:teuthology.orchestra.run.smithi142.stderr: 93.4% -- replaced with /var/log/ceph/cluster2.audit.log.gz
2024-05-08T01:37:56.833 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.70935.log:	/var/log/ceph/cluster1-client.admin.29578.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29578.log.gz
2024-05-08T01:37:56.833 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70935.log.gz
2024-05-08T01:37:56.833 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35842.log
2024-05-08T01:37:56.834 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2.log.gz
2024-05-08T01:37:56.834 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.53069.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71695.log
2024-05-08T01:37:56.834 INFO:teuthology.orchestra.run.smithi142.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.31313.log.gz
2024-05-08T01:37:56.834 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.35842.log:	gzip -5 88.1% --verbose -- -- replaced with /var/log/ceph/cluster2-client.admin.53069.log.gz /var/log/ceph/cluster1-client.admin.33150.log
2024-05-08T01:37:56.834 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:56.834 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.35842.log.gz
2024-05-08T01:37:56.834 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.71695.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68331.log
2024-05-08T01:37:56.834 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34519.log
2024-05-08T01:37:56.834 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71695.log.gz
2024-05-08T01:37:56.834 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.33150.log:	/var/log/ceph/cluster1-client.mirror.68331.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73973.log
2024-05-08T01:37:56.834 INFO:teuthology.orchestra.run.smithi142.stderr: 86.7% 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.33150.log.gz
2024-05-08T01:37:56.834 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.68331.log.gz
2024-05-08T01:37:56.834 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40627.log
2024-05-08T01:37:56.835 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.34519.log:	/var/log/ceph/cluster1-client.mirror.73973.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73973.log.gz
2024-05-08T01:37:56.835 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47344.log
2024-05-08T01:37:56.835 INFO:teuthology.orchestra.run.smithi142.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.34519.log.gz
2024-05-08T01:37:56.908 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34615.log
2024-05-08T01:37:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.40627.log:	/var/log/ceph/cluster2-client.admin.47344.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42225.log
2024-05-08T01:37:56.909 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.40627.log.gz
2024-05-08T01:37:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.34615.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39267.log
2024-05-08T01:37:56.909 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.47344.log.gz
2024-05-08T01:37:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.42225.log:	 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.34615.log.gz
2024-05-08T01:37:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41014.log
2024-05-08T01:37:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.39267.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.30266.log
2024-05-08T01:37:56.909 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.42225.log.gz
2024-05-08T01:37:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.41014.log:	 94.5% -- replaced with /var/log/ceph/cluster2-client.mirror.2.35274.log.gz
2024-05-08T01:37:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63504.log
2024-05-08T01:37:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.0.30266.log:	 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.41014.log.gz
2024-05-08T01:37:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45067.log
2024-05-08T01:37:56.909 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39267.log.gz
2024-05-08T01:37:56.909 INFO:teuthology.orchestra.run.smithi142.stderr: 19.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.30266.log.gz
2024-05-08T01:37:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.63504.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31086.log
2024-05-08T01:37:56.910 INFO:teuthology.orchestra.run.smithi142.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.admin.63504.log.gz
2024-05-08T01:37:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.45067.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31956.log
2024-05-08T01:37:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.31086.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.45067.log.gz
2024-05-08T01:37:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60971.log
2024-05-08T01:37:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.31956.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39632.log
2024-05-08T01:37:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.60971.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39897.log
2024-05-08T01:37:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.39632.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56206.log
2024-05-08T01:37:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.39897.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29921.log
2024-05-08T01:37:56.910 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.39632.log.gz
2024-05-08T01:37:56.910 INFO:teuthology.orchestra.run.smithi142.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.31086.log.gz
2024-05-08T01:37:56.910 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.39897.log.gz
2024-05-08T01:37:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.56206.log:	 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.31956.log.gz
2024-05-08T01:37:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34486.log
2024-05-08T01:37:56.911 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.56206.log.gz
2024-05-08T01:37:56.911 INFO:teuthology.orchestra.run.smithi142.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60971.log.gz
2024-05-08T01:37:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.29921.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71487.log
2024-05-08T01:37:56.911 INFO:teuthology.orchestra.run.smithi142.stderr: 54.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29921.log.gz
2024-05-08T01:37:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.34486.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59702.log
2024-05-08T01:37:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.71487.log:	 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.34486.log.gz
2024-05-08T01:37:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31757.log
2024-05-08T01:37:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34793.log
2024-05-08T01:37:56.911 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71487.log.gz
2024-05-08T01:37:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.59702.log:	/var/log/ceph/cluster2-client.mirror.31757.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49457.log
2024-05-08T01:37:56.911 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59702.log.gz
2024-05-08T01:37:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29603.log
2024-05-08T01:37:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.34793.log:	 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.31757.log.gz
2024-05-08T01:37:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34793.log.gz
2024-05-08T01:37:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.49457.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51343.log
2024-05-08T01:37:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.29603.log:	 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49457.log.gz
2024-05-08T01:37:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29603.log.gz
2024-05-08T01:37:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73651.log
2024-05-08T01:37:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.51343.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29822.log
2024-05-08T01:37:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.73651.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60950.log
2024-05-08T01:37:56.912 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73651.log.gz
2024-05-08T01:37:56.912 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.51343.log.gz
2024-05-08T01:37:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.29822.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74521.log
2024-05-08T01:37:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29822.log.gz
2024-05-08T01:37:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.60950.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56631.log
2024-05-08T01:37:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.74521.log:	  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74521.log.gz
2024-05-08T01:37:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40598.log
2024-05-08T01:37:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58124.log
2024-05-08T01:37:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.56631.log:	 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60950.log.gz
2024-05-08T01:37:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.40598.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64535.log
2024-05-08T01:37:56.913 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.56631.log.gz
2024-05-08T01:37:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58124.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44081.log
2024-05-08T01:37:56.913 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.40598.log.gz
2024-05-08T01:37:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.64535.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61442.log
2024-05-08T01:37:56.913 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58124.log.gz
2024-05-08T01:37:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44693.log
2024-05-08T01:37:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.44081.log:	 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64535.log.gz
2024-05-08T01:37:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.61442.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50210.log
2024-05-08T01:37:56.913 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.44081.log.gz
2024-05-08T01:37:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.44693.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57389.log
2024-05-08T01:37:56.913 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.61442.log.gz
2024-05-08T01:37:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.50210.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53184.log
2024-05-08T01:37:56.914 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.44693.log.gz
2024-05-08T01:37:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.57389.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74050.log
2024-05-08T01:37:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.53184.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55678.log
2024-05-08T01:37:56.914 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.57389.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.50210.log.gz
2024-05-08T01:37:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.74050.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34462.log
2024-05-08T01:37:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74050.log.gz
2024-05-08T01:37:56.914 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.53184.log.gz
2024-05-08T01:37:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54355.log
2024-05-08T01:37:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.55678.log:	/var/log/ceph/cluster2-client.mirror.34462.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63290.log
2024-05-08T01:37:56.914 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.55678.log.gz
2024-05-08T01:37:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.54355.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72437.log
2024-05-08T01:37:56.914 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.54355.log.gz
2024-05-08T01:37:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.63290.log:	 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.34462.log.gz
2024-05-08T01:37:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74223.log
2024-05-08T01:37:56.915 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63290.log.gz
2024-05-08T01:37:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32870.log
2024-05-08T01:37:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.72437.log:	/var/log/ceph/cluster2-client.admin.74223.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39008.log
2024-05-08T01:37:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74223.log.gz
2024-05-08T01:37:56.915 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72437.log.gz
2024-05-08T01:37:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.32870.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30066.log
2024-05-08T01:37:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.39008.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.32870.log.gz
2024-05-08T01:37:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54083.log
2024-05-08T01:37:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.30066.log:	  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30066.log.gz
2024-05-08T01:37:56.915 INFO:teuthology.orchestra.run.smithi142.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39008.log.gzgzip
2024-05-08T01:37:56.915 INFO:teuthology.orchestra.run.smithi142.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58486.log
2024-05-08T01:37:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69293.log
2024-05-08T01:37:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.54083.log:	/var/log/ceph/cluster2-client.mirror.58486.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43145.log
2024-05-08T01:37:56.916 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5%/var/log/ceph/cluster1-client.mirror.69293.log:	 -- replaced with /var/log/ceph/cluster1-client.admin.54083.log.gz
2024-05-08T01:37:56.916 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.58486.log.gz
2024-05-08T01:37:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36358.log
2024-05-08T01:37:56.916 INFO:teuthology.orchestra.run.smithi142.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69293.log.gz
2024-05-08T01:37:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.43145.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70695.log
2024-05-08T01:37:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.36358.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66104.log
2024-05-08T01:37:56.916 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.43145.log.gz
2024-05-08T01:37:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.70695.log:	 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.36358.log.gz
2024-05-08T01:37:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59680.log
2024-05-08T01:37:56.916 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.70695.log.gz
2024-05-08T01:37:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66360.log
2024-05-08T01:37:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.66104.log:	/var/log/ceph/cluster2-client.mirror.59680.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68590.log
2024-05-08T01:37:56.916 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66104.log.gz
2024-05-08T01:37:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.66360.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34693.log
2024-05-08T01:37:56.917 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59680.log.gz
2024-05-08T01:37:56.917 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.68590.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34540.log
2024-05-08T01:37:56.917 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43725.log
2024-05-08T01:37:56.917 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4%/var/log/ceph/cluster1-client.mirror.34693.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.68590.log.gz
2024-05-08T01:37:56.917 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.34693.log.gz
2024-05-08T01:37:56.917 INFO:teuthology.orchestra.run.smithi142.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66360.log.gz
2024-05-08T01:37:56.917 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.34540.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67992.log
2024-05-08T01:37:56.917 INFO:teuthology.orchestra.run.smithi142.stderr: 81.1% -- replaced with /var/log/ceph/cluster2-client.mirror.34540.log.gz
2024-05-08T01:37:56.917 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48685.log
2024-05-08T01:37:56.917 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.43725.log:	/var/log/ceph/cluster1-client.admin.67992.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64323.log
2024-05-08T01:37:56.917 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.43725.log.gz
2024-05-08T01:37:56.917 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.48685.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59755.log
2024-05-08T01:37:56.917 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.67992.log.gz
2024-05-08T01:37:56.917 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.64323.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49925.log
2024-05-08T01:37:56.918 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.48685.log.gz
2024-05-08T01:37:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.59755.log:	gzip -5 --verbose -- 87.4% /var/log/ceph/cluster2-client.admin.39414.log -- replaced with /var/log/ceph/cluster2-client.mirror.64323.log.gz
2024-05-08T01:37:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.49925.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43952.log
2024-05-08T01:37:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.39414.log:	gzip 87.4% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.49925.log.gz --verbose
2024-05-08T01:37:56.918 INFO:teuthology.orchestra.run.smithi142.stderr: -- /var/log/ceph/cluster1-client.mirror.37304.log
2024-05-08T01:37:56.918 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59755.log.gz
2024-05-08T01:37:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.43952.log:	 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.39414.log.gz
2024-05-08T01:37:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45308.log
2024-05-08T01:37:56.918 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43952.log.gz/var/log/ceph/cluster1-client.mirror.37304.log:
2024-05-08T01:37:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57099.log
2024-05-08T01:37:56.918 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.37304.log.gz
2024-05-08T01:37:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52047.log
2024-05-08T01:37:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.45308.log:	/var/log/ceph/cluster2-client.admin.57099.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48830.log
2024-05-08T01:37:56.919 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45308.log.gz
2024-05-08T01:37:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.52047.log:	 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.57099.log.gz
2024-05-08T01:37:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30228.log
2024-05-08T01:37:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.48830.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55210.log
2024-05-08T01:37:56.919 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52047.log.gz
2024-05-08T01:37:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.30228.log:	gzip -5 88.1% --verbose -- replaced with /var/log/ceph/cluster1-client.admin.48830.log.gz --
2024-05-08T01:37:56.919 INFO:teuthology.orchestra.run.smithi142.stderr: /var/log/ceph/cluster2-client.mirror.62925.log
2024-05-08T01:37:56.919 INFO:teuthology.orchestra.run.smithi142.stderr: 55.6% -- replaced with /var/log/ceph/cluster2-client.mirror.30228.log.gz
2024-05-08T01:37:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47180.log
2024-05-08T01:37:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.55210.log:	/var/log/ceph/cluster2-client.mirror.62925.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71823.log
2024-05-08T01:37:56.919 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.55210.log.gz
2024-05-08T01:37:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.47180.log:	 88.0%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.62925.log.gz
2024-05-08T01:37:56.919 INFO:teuthology.orchestra.run.smithi142.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70124.log
2024-05-08T01:37:56.919 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47180.log.gz
2024-05-08T01:37:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.71823.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51463.log
2024-05-08T01:37:56.920 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71823.log.gz
2024-05-08T01:37:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51017.log
2024-05-08T01:37:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.70124.log:	/var/log/ceph/cluster2-client.admin.51463.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.73607.log
2024-05-08T01:37:56.920 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70124.log.gz
2024-05-08T01:37:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.51017.log:	 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.51463.log.gz
2024-05-08T01:37:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71258.log
2024-05-08T01:37:56.920 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3%/var/log/ceph/cluster2-client.mirror.1.73607.log:	 -- replaced with /var/log/ceph/cluster1-client.mirror.51017.log.gz
2024-05-08T01:37:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61191.log
2024-05-08T01:37:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67061.log
2024-05-08T01:37:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.71258.log:	 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71258.log.gz
2024-05-08T01:37:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.61191.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61383.log
2024-05-08T01:37:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56512.log
2024-05-08T01:37:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.67061.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61191.log.gz
2024-05-08T01:37:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.61383.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54728.log
2024-05-08T01:37:56.921 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.67061.log.gz
2024-05-08T01:37:56.921 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.56512.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35161.log
2024-05-08T01:37:56.921 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61383.log.gz
2024-05-08T01:37:56.921 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.54728.log:	 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.1.73607.log.gz
2024-05-08T01:37:56.921 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.56512.log.gz
2024-05-08T01:37:56.921 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51976.log
2024-05-08T01:37:56.921 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.35161.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70664.log
2024-05-08T01:37:56.921 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.54728.log.gz
2024-05-08T01:37:56.921 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.51976.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40200.log
2024-05-08T01:37:56.921 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.35161.log.gz
2024-05-08T01:37:56.921 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.70664.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45675.log
2024-05-08T01:37:56.921 INFO:teuthology.orchestra.run.smithi142.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51976.log.gz
2024-05-08T01:37:56.921 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.40200.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55421.log
2024-05-08T01:37:56.921 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.70664.log.gz
2024-05-08T01:37:56.922 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.45675.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70324.log
2024-05-08T01:37:56.922 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.55421.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53244.log
2024-05-08T01:37:56.922 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40200.log.gz
2024-05-08T01:37:56.922 INFO:teuthology.orchestra.run.smithi142.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45675.log.gz
2024-05-08T01:37:56.922 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.70324.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65928.log
2024-05-08T01:37:56.922 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.55421.log.gz
2024-05-08T01:37:56.922 INFO:teuthology.orchestra.run.smithi142.stderr:gzip/var/log/ceph/cluster2-client.admin.53244.log:	 -5 --verbose -- /var/log/ceph/cluster2-client.admin.41802.log
2024-05-08T01:37:56.922 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70324.log.gz
2024-05-08T01:37:56.922 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.65928.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35974.log
2024-05-08T01:37:56.922 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.53244.log.gz
2024-05-08T01:37:56.922 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.41802.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50755.log
2024-05-08T01:37:56.922 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4%/var/log/ceph/cluster2-client.mirror.35974.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.65928.log.gz
2024-05-08T01:37:56.922 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70177.log
2024-05-08T01:37:56.922 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.41802.log.gz
2024-05-08T01:37:56.923 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.35974.log.gz
2024-05-08T01:37:56.923 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.50755.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72362.log
2024-05-08T01:37:56.923 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35694.log
2024-05-08T01:37:56.923 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.70177.log:	 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.50755.log.gz
2024-05-08T01:37:56.923 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.72362.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50903.log
2024-05-08T01:37:56.923 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70177.log.gz
2024-05-08T01:37:56.923 INFO:teuthology.orchestra.run.smithi142.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72362.log.gz
2024-05-08T01:37:56.923 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.35694.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68979.log
2024-05-08T01:37:56.923 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68934.log
2024-05-08T01:37:56.923 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.50903.log:	 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.35694.log.gz
2024-05-08T01:37:56.923 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.68979.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51494.log
2024-05-08T01:37:56.923 INFO:teuthology.orchestra.run.smithi142.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68979.log.gz
2024-05-08T01:37:56.923 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.50903.log.gz
2024-05-08T01:37:56.923 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.68934.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48005.log
2024-05-08T01:37:56.924 INFO:teuthology.orchestra.run.smithi142.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68934.log.gz
2024-05-08T01:37:56.924 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.51494.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59634.log
2024-05-08T01:37:56.924 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.48005.log:	 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.51494.log.gz
2024-05-08T01:37:56.924 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47632.log
2024-05-08T01:37:56.924 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.59634.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68271.log
2024-05-08T01:37:56.924 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.48005.log.gz
2024-05-08T01:37:56.924 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.47632.log:	gzip 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59634.log.gz -5
2024-05-08T01:37:56.924 INFO:teuthology.orchestra.run.smithi142.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.30006.log
2024-05-08T01:37:56.924 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47632.log.gz
2024-05-08T01:37:56.924 INFO:teuthology.orchestra.run.smithi142.stderr:gzip/var/log/ceph/cluster1-client.mirror.68271.log:	 -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56355.log
2024-05-08T01:37:56.924 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.30006.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42165.log
2024-05-08T01:37:56.924 INFO:teuthology.orchestra.run.smithi142.stderr: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.30006.log.gz
2024-05-08T01:37:56.924 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68271.log.gz
2024-05-08T01:37:56.924 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.56355.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70555.log
2024-05-08T01:37:56.924 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56355.log.gz
2024-05-08T01:37:56.925 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.42165.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73288.log
2024-05-08T01:37:56.925 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65795.log
2024-05-08T01:37:56.925 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.70555.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.42165.log.gz
2024-05-08T01:37:56.925 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.73288.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58696.log
2024-05-08T01:37:56.925 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70555.log.gz
2024-05-08T01:37:56.925 INFO:teuthology.orchestra.run.smithi142.stderr:gzip/var/log/ceph/cluster2-client.mirror.65795.log:	 -5 --verbose 87.8% -- /var/log/ceph/cluster1-client.admin.32693.log -- replaced with /var/log/ceph/cluster1-client.admin.73288.log.gz
2024-05-08T01:37:56.925 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:56.925 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58696.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63408.log
2024-05-08T01:37:56.925 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.32693.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52358.log
2024-05-08T01:37:56.925 INFO:teuthology.orchestra.run.smithi142.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65795.log.gz
2024-05-08T01:37:56.925 INFO:teuthology.orchestra.run.smithi142.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58696.log.gz
2024-05-08T01:37:56.925 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.63408.log:	gzip -5 --verbose 88.2% -- -- replaced with /var/log/ceph/cluster1-client.admin.32693.log.gz /var/log/ceph/cluster2-client.admin.51709.log
2024-05-08T01:37:56.925 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:56.926 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63408.log.gz
2024-05-08T01:37:56.926 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.52358.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36765.log
2024-05-08T01:37:56.926 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.51709.log:	 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.52358.log.gz
2024-05-08T01:37:56.926 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63060.log
2024-05-08T01:37:56.926 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56116.log
2024-05-08T01:37:56.926 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.51709.log.gz
2024-05-08T01:37:56.926 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.36765.log:	/var/log/ceph/cluster1-client.mirror.63060.log:	 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36765.log.gz
2024-05-08T01:37:56.926 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56481.log
2024-05-08T01:37:56.926 INFO:teuthology.orchestra.run.smithi142.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63060.log.gz
2024-05-08T01:37:56.926 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73140.log
2024-05-08T01:37:56.926 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.56116.log:	/var/log/ceph/cluster1-client.admin.56481.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71647.log
2024-05-08T01:37:56.926 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.56116.log.gz
2024-05-08T01:37:56.926 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58076.log
2024-05-08T01:37:56.927 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.73140.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.56481.log.gz
2024-05-08T01:37:56.927 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.71647.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57441.log
2024-05-08T01:37:56.927 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.73140.log.gz
2024-05-08T01:37:56.927 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.58076.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41830.log
2024-05-08T01:37:56.927 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71647.log.gz
2024-05-08T01:37:56.927 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.57441.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58076.log.gz
2024-05-08T01:37:56.927 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51612.log
2024-05-08T01:37:56.927 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48949.log
2024-05-08T01:37:56.927 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.41830.log:	/var/log/ceph/cluster2-client.mirror.51612.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.69775.log
2024-05-08T01:37:56.927 INFO:teuthology.orchestra.run.smithi142.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41830.log.gz
2024-05-08T01:37:56.927 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51612.log.gz
2024-05-08T01:37:56.927 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.48949.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55742.log
2024-05-08T01:37:56.927 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.3.69775.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30926.log
2024-05-08T01:37:56.927 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48949.log.gz
2024-05-08T01:37:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74703.log
2024-05-08T01:37:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.55742.log:	/var/log/ceph/cluster2-client.mirror.30926.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43085.log
2024-05-08T01:37:56.928 INFO:teuthology.orchestra.run.smithi142.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.admin.55742.log.gz
2024-05-08T01:37:56.928 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6%/var/log/ceph/cluster2-client.admin.74703.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.30926.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61092.log
2024-05-08T01:37:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74703.log.gz
2024-05-08T01:37:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65628.log
2024-05-08T01:37:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.43085.log:	/var/log/ceph/cluster2-client.mirror.61092.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33243.log
2024-05-08T01:37:56.928 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.43085.log.gz
2024-05-08T01:37:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.65628.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48552.log
2024-05-08T01:37:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37750.log
2024-05-08T01:37:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.33243.log:	 89.5%/var/log/ceph/cluster2-client.mirror.48552.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.65628.log.gz
2024-05-08T01:37:56.928 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48552.log.gz
2024-05-08T01:37:56.928 INFO:teuthology.orchestra.run.smithi142.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.33243.log.gz
2024-05-08T01:37:56.929 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43310.log
2024-05-08T01:37:56.929 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.37750.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31128.log
2024-05-08T01:37:56.929 INFO:teuthology.orchestra.run.smithi142.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.3.69775.log.gz
2024-05-08T01:37:56.929 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.43310.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66649.log
2024-05-08T01:37:56.929 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.37750.log.gz
2024-05-08T01:37:56.929 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59508.log
2024-05-08T01:37:56.929 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43310.log.gz
2024-05-08T01:37:56.929 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.31128.log:	/var/log/ceph/cluster1-client.mirror.66649.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59891.log
2024-05-08T01:37:56.929 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.59508.log:	 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.66649.log.gz
2024-05-08T01:37:56.929 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38829.log
2024-05-08T01:37:56.929 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.31128.log.gz
2024-05-08T01:37:56.929 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.59891.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37841.log
2024-05-08T01:37:56.929 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59508.log.gz
2024-05-08T01:37:56.929 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59891.log.gz
2024-05-08T01:37:56.930 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.38829.log:	 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38829.log.gz
2024-05-08T01:37:56.930 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42077.log
2024-05-08T01:37:56.930 INFO:teuthology.orchestra.run.smithi142.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.57441.log.gz
2024-05-08T01:37:56.930 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.37841.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50018.log
2024-05-08T01:37:56.930 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.42077.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50470.log
2024-05-08T01:37:56.930 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.37841.log.gz
2024-05-08T01:37:56.930 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.50018.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71432.log
2024-05-08T01:37:56.930 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.42077.log.gz
2024-05-08T01:37:56.930 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.50470.log:	 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50018.log.gz
2024-05-08T01:37:56.930 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29403.log
2024-05-08T01:37:56.930 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50470.log.gz
2024-05-08T01:37:56.930 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.71432.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57641.log
2024-05-08T01:37:56.930 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.29403.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61331.log
2024-05-08T01:37:56.930 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29403.log.gz
2024-05-08T01:37:56.931 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71432.log.gz
2024-05-08T01:37:56.931 INFO:teuthology.orchestra.run.smithi142.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61092.log.gz
2024-05-08T01:37:56.931 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.57641.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69352.log
2024-05-08T01:37:56.931 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.61331.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56540.log
2024-05-08T01:37:56.931 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57641.log.gz
2024-05-08T01:37:56.931 INFO:teuthology.orchestra.run.smithi142.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61331.log.gz
2024-05-08T01:37:56.931 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.69352.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34149.log
2024-05-08T01:37:56.931 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69352.log.gz
2024-05-08T01:37:56.931 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.56540.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62724.log
2024-05-08T01:37:56.931 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61487.log
2024-05-08T01:37:56.931 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.34149.log:	/var/log/ceph/cluster1-client.mirror.62724.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29553.log
2024-05-08T01:37:56.931 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.56540.log.gz
2024-05-08T01:37:56.931 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62724.log.gz
2024-05-08T01:37:56.931 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.61487.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54012.log
2024-05-08T01:37:56.932 INFO:teuthology.orchestra.run.smithi142.stderr:gzip/var/log/ceph/cluster1-client.admin.29553.log:	 -5 --verbose -- /var/log/ceph/cluster2-client.admin.29181.log
2024-05-08T01:37:56.932 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29553.log.gz
2024-05-08T01:37:56.932 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.54012.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36051.log
2024-05-08T01:37:56.932 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.29181.log:	 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54012.log.gz
2024-05-08T01:37:56.932 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.66816.log
2024-05-08T01:37:56.932 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29181.log.gz
2024-05-08T01:37:56.932 INFO:teuthology.orchestra.run.smithi142.stderr: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34149.log.gz
2024-05-08T01:37:56.932 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.36051.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44485.log
2024-05-08T01:37:56.932 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.3.66816.log:	 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36051.log.gz
2024-05-08T01:37:56.932 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73457.log
2024-05-08T01:37:56.932 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.44485.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30194.log
2024-05-08T01:37:56.932 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.73457.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58096.log
2024-05-08T01:37:56.932 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.44485.log.gz
2024-05-08T01:37:56.932 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.30194.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40427.log
2024-05-08T01:37:56.932 INFO:teuthology.orchestra.run.smithi142.stderr: 64.7% -- replaced with /var/log/ceph/cluster2-client.mirror.30194.log.gz 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.73457.log.gz
2024-05-08T01:37:56.933 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:56.933 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70075.log
2024-05-08T01:37:56.933 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.58096.log:	/var/log/ceph/cluster1-client.mirror.40427.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56176.log
2024-05-08T01:37:56.933 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.70075.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47005.log
2024-05-08T01:37:56.933 INFO:teuthology.orchestra.run.smithi142.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40427.log.gz
2024-05-08T01:37:56.933 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.56176.log:	 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70075.log.gz
2024-05-08T01:37:56.933 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.56176.log.gz
2024-05-08T01:37:56.933 INFO:teuthology.orchestra.run.smithi142.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58096.log.gz
2024-05-08T01:37:56.933 INFO:teuthology.orchestra.run.smithi142.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.3.66816.log.gz
2024-05-08T01:37:56.933 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42981.log
2024-05-08T01:37:56.933 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.47005.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50122.log
2024-05-08T01:37:56.933 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.42981.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56719.log
2024-05-08T01:37:56.933 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.47005.log.gz
2024-05-08T01:37:56.933 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.50122.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65994.log
2024-05-08T01:37:56.934 INFO:teuthology.orchestra.run.smithi142.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42981.log.gz
2024-05-08T01:37:56.934 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.56719.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61668.log
2024-05-08T01:37:56.934 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.50122.log.gz
2024-05-08T01:37:56.934 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44420.log
2024-05-08T01:37:56.934 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.65994.log:	 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56719.log.gz
2024-05-08T01:37:56.934 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.61668.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.24067.log
2024-05-08T01:37:56.934 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61668.log.gz
2024-05-08T01:37:56.934 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.44420.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52210.log
2024-05-08T01:37:56.934 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65994.log.gz
2024-05-08T01:37:56.934 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44420.log.gz
2024-05-08T01:37:56.934 INFO:teuthology.orchestra.run.smithi142.stderr: 94.1%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.61487.log.gz -5
2024-05-08T01:37:56.934 INFO:teuthology.orchestra.run.smithi142.stderr: --verbose --/var/log/ceph/cluster2-client.admin.24067.log:	 /var/log/ceph/cluster1-client.mirror.62618.log
2024-05-08T01:37:56.934 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.24067.log.gz
2024-05-08T01:37:56.934 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.52210.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55066.log
2024-05-08T01:37:56.935 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.62618.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58144.log
2024-05-08T01:37:56.935 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.52210.log.gz
2024-05-08T01:37:56.935 INFO:teuthology.orchestra.run.smithi142.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62618.log.gz
2024-05-08T01:37:56.935 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.55066.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37583.log
2024-05-08T01:37:56.935 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.55066.log.gz
2024-05-08T01:37:56.935 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58144.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29678.log
2024-05-08T01:37:56.935 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63197.log
2024-05-08T01:37:56.935 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.37583.log:	/var/log/ceph/cluster2-client.admin.29678.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.69421.log
2024-05-08T01:37:56.935 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29678.log.gz
2024-05-08T01:37:56.935 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.37583.log.gz
2024-05-08T01:37:56.935 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.63197.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46762.log
2024-05-08T01:37:56.935 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63197.log.gz
2024-05-08T01:37:56.935 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.69421.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44573.log
2024-05-08T01:37:56.935 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37962.log
2024-05-08T01:37:56.936 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.46762.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.69421.log.gz
2024-05-08T01:37:56.936 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.44573.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39110.log
2024-05-08T01:37:56.936 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46762.log.gz
2024-05-08T01:37:56.936 INFO:teuthology.orchestra.run.smithi142.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58144.log.gz
2024-05-08T01:37:56.936 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.37962.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65861.log
2024-05-08T01:37:56.936 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.39110.log:	 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37962.log.gz
2024-05-08T01:37:56.936 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62024.log
2024-05-08T01:37:56.936 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44573.log.gz
2024-05-08T01:37:56.936 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.65861.log:	 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39110.log.gzgzip
2024-05-08T01:37:56.936 INFO:teuthology.orchestra.run.smithi142.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68824.log
2024-05-08T01:37:56.936 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.62024.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71289.log
2024-05-08T01:37:56.936 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65861.log.gz
2024-05-08T01:37:56.936 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.68824.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45220.log
2024-05-08T01:37:56.936 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.62024.log.gz
2024-05-08T01:37:56.937 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.71289.log:	 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68824.log.gz
2024-05-08T01:37:56.937 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71111.log
2024-05-08T01:37:56.937 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29177.log
2024-05-08T01:37:56.937 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.45220.log:	/var/log/ceph/cluster1-client.mirror.71111.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32001.log
2024-05-08T01:37:56.937 INFO:teuthology.orchestra.run.smithi142.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71289.log.gz
2024-05-08T01:37:56.937 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45220.log.gz
2024-05-08T01:37:56.937 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.29177.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61551.log
2024-05-08T01:37:56.937 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29177.log.gz
2024-05-08T01:37:56.937 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.32001.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38203.log
2024-05-08T01:37:56.937 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.61551.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33212.log
2024-05-08T01:37:56.937 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.38203.log:	 92.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37523.log
2024-05-08T01:37:56.937 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.71111.log.gz
2024-05-08T01:37:56.937 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.33212.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.24136.log
2024-05-08T01:37:56.937 INFO:teuthology.orchestra.run.smithi142.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.61551.log.gz
2024-05-08T01:37:56.937 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.38203.log.gz
2024-05-08T01:37:56.938 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.33212.log.gz
2024-05-08T01:37:56.938 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.37523.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.37523.log.gz
2024-05-08T01:37:56.938 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63232.log
2024-05-08T01:37:56.938 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.24136.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52325.log
2024-05-08T01:37:56.938 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.24136.log.gz
2024-05-08T01:37:56.938 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.63232.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63037.log
2024-05-08T01:37:56.938 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.52325.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.35235.log
2024-05-08T01:37:56.938 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63232.log.gz
2024-05-08T01:37:56.938 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.63037.log:	 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.52325.log.gz
2024-05-08T01:37:56.938 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73395.log
2024-05-08T01:37:56.938 INFO:teuthology.orchestra.run.smithi142.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63037.log.gz
2024-05-08T01:37:56.938 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65651.log
2024-05-08T01:37:56.938 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.2.35235.log:	 19.9% -- replaced with /var/log/ceph/cluster2-client.mirror.2.35235.log.gz
2024-05-08T01:37:56.938 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.73395.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62239.log
2024-05-08T01:37:56.939 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.65651.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.73395.log.gz
2024-05-08T01:37:56.939 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.69094.log
2024-05-08T01:37:56.939 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34970.log
2024-05-08T01:37:56.939 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.62239.log:	/var/log/ceph/cluster1-client.admin.69094.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28946.log
2024-05-08T01:37:56.939 INFO:teuthology.orchestra.run.smithi142.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65651.log.gz
2024-05-08T01:37:56.939 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.34970.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66684.log
2024-05-08T01:37:56.939 INFO:teuthology.orchestra.run.smithi142.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62239.log.gz
2024-05-08T01:37:56.939 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.69094.log.gz
2024-05-08T01:37:56.939 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.34970.log.gz
2024-05-08T01:37:56.939 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.28946.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62963.log
2024-05-08T01:37:56.939 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28946.log.gz
2024-05-08T01:37:56.939 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.66684.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64444.log
2024-05-08T01:37:56.939 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.62963.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68107.log
2024-05-08T01:37:56.939 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.64444.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34843.log
2024-05-08T01:37:56.940 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64444.log.gz
2024-05-08T01:37:56.940 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.68107.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38377.log
2024-05-08T01:37:56.940 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.34843.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71896.log
2024-05-08T01:37:56.940 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34843.log.gz
2024-05-08T01:37:56.940 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.38377.log:	 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62963.log.gz
2024-05-08T01:37:56.940 INFO:teuthology.orchestra.run.smithi142.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32001.log.gz
2024-05-08T01:37:56.940 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58350.log
2024-05-08T01:37:56.940 INFO:teuthology.orchestra.run.smithi142.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38377.log.gz
2024-05-08T01:37:56.940 INFO:teuthology.orchestra.run.smithi142.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68107.log.gz
2024-05-08T01:37:56.940 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.71896.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48171.log
2024-05-08T01:37:56.940 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58350.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38741.log
2024-05-08T01:37:56.940 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.48171.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58327.log
2024-05-08T01:37:56.940 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58350.log.gz
2024-05-08T01:37:56.940 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.38741.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29328.log
2024-05-08T01:37:56.940 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.48171.log.gz
2024-05-08T01:37:56.941 INFO:teuthology.orchestra.run.smithi142.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71896.log.gz
2024-05-08T01:37:56.941 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9%/var/log/ceph/cluster2-client.mirror.58327.log:	 -- replaced with /var/log/ceph/cluster1-client.admin.38741.log.gz
2024-05-08T01:37:56.941 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35630.log
2024-05-08T01:37:56.941 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.29328.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29328.log.gz
2024-05-08T01:37:56.941 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.69729.log
2024-05-08T01:37:57.067 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52484.log
2024-05-08T01:37:57.067 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.35630.log:	/var/log/ceph/cluster1-client.mirror.3.69729.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56660.log
2024-05-08T01:37:57.068 INFO:teuthology.orchestra.run.smithi142.stderr: 20.5% -- replaced with /var/log/ceph/cluster1-client.mirror.3.69729.log.gz
2024-05-08T01:37:57.068 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.35630.log.gz
2024-05-08T01:37:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.52484.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41885.log
2024-05-08T01:37:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.56660.log:	 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52484.log.gz
2024-05-08T01:37:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51948.log
2024-05-08T01:37:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.41885.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38641.log
2024-05-08T01:37:57.068 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.56660.log.gz
2024-05-08T01:37:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.51948.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45730.log
2024-05-08T01:37:57.068 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41885.log.gz
2024-05-08T01:37:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.38641.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68153.log
2024-05-08T01:37:57.068 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.51948.log.gz
2024-05-08T01:37:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59464.log
2024-05-08T01:37:57.068 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.38641.log.gz
2024-05-08T01:37:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.45730.log:	/var/log/ceph/cluster2-client.mirror.68153.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66038.log
2024-05-08T01:37:57.069 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5%/var/log/ceph/cluster2-client.mirror.59464.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.45730.log.gz
2024-05-08T01:37:57.069 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68153.log.gz
2024-05-08T01:37:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71091.log
2024-05-08T01:37:57.069 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59464.log.gz
2024-05-08T01:37:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39925.log
2024-05-08T01:37:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.66038.log:	/var/log/ceph/cluster2-client.mirror.71091.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49870.log
2024-05-08T01:37:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.39925.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59821.log
2024-05-08T01:37:57.069 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66038.log.gz
2024-05-08T01:37:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.49870.log:	 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39925.log.gz
2024-05-08T01:37:57.069 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.49870.log.gz
2024-05-08T01:37:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62340.log
2024-05-08T01:37:57.069 INFO:teuthology.orchestra.run.smithi142.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71091.log.gz
2024-05-08T01:37:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.59821.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63533.log
2024-05-08T01:37:57.070 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59821.log.gz
2024-05-08T01:37:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.62340.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39146.log
2024-05-08T01:37:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53423.log
2024-05-08T01:37:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.63533.log:	 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62340.log.gz
2024-05-08T01:37:57.070 INFO:teuthology.orchestra.run.smithi142.stderr: 63.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63533.log.gz
2024-05-08T01:37:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.39146.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50505.log
2024-05-08T01:37:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.53423.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67569.log
2024-05-08T01:37:57.070 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.39146.log.gz
2024-05-08T01:37:57.070 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53423.log.gz
2024-05-08T01:37:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.50505.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31824.log
2024-05-08T01:37:57.070 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50505.log.gz
2024-05-08T01:37:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.67569.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73201.log
2024-05-08T01:37:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.31824.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46453.log
2024-05-08T01:37:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.73201.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.24184.log
2024-05-08T01:37:57.071 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.31824.log.gz
2024-05-08T01:37:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.46453.log:	 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.46453.log.gz
2024-05-08T01:37:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62440.log
2024-05-08T01:37:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.24184.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64355.log
2024-05-08T01:37:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.24184.log.gz 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73201.log.gz
2024-05-08T01:37:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.62440.log:	 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62440.log.gz
2024-05-08T01:37:57.071 INFO:teuthology.orchestra.run.smithi142.stderr: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28913.log
2024-05-08T01:37:57.071 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.58327.log.gz
2024-05-08T01:37:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.64355.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68746.log
2024-05-08T01:37:57.071 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.64355.log.gz
2024-05-08T01:37:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.28913.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63383.log
2024-05-08T01:37:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28913.log.gz
2024-05-08T01:37:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.68746.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62485.log
2024-05-08T01:37:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.63383.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46660.log
2024-05-08T01:37:57.072 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.68746.log.gz
2024-05-08T01:37:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.62485.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62574.log
2024-05-08T01:37:57.072 INFO:teuthology.orchestra.run.smithi142.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62485.log.gz
2024-05-08T01:37:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48264.log
2024-05-08T01:37:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.46660.log:	/var/log/ceph/cluster2-client.mirror.62574.log:	 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62574.log.gz
2024-05-08T01:37:57.072 INFO:teuthology.orchestra.run.smithi142.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46660.log.gz
2024-05-08T01:37:57.072 INFO:teuthology.orchestra.run.smithi142.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63383.log.gz
2024-05-08T01:37:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45128.log
2024-05-08T01:37:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.48264.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37552.log
2024-05-08T01:37:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.45128.log:	 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.48264.log.gz
2024-05-08T01:37:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57739.log
2024-05-08T01:37:57.073 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.37552.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30406.log
2024-05-08T01:37:57.073 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.45128.log.gz
2024-05-08T01:37:57.073 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.57739.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57676.log
2024-05-08T01:37:57.073 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.37552.log.gz
2024-05-08T01:37:57.073 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.24160.log
2024-05-08T01:37:57.073 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.30406.log:	 82.4%/var/log/ceph/cluster2-client.mirror.57676.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.30406.log.gz 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67569.log.gz
2024-05-08T01:37:57.073 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:57.073 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50089.log
2024-05-08T01:37:57.073 INFO:teuthology.orchestra.run.smithi142.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.57739.log.gz
2024-05-08T01:37:57.073 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57676.log.gz
2024-05-08T01:37:57.073 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.24160.log:	  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.24160.log.gz
2024-05-08T01:37:57.073 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44452.log
2024-05-08T01:37:57.073 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.50089.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58216.log
2024-05-08T01:37:57.073 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.50089.log.gz
2024-05-08T01:37:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.44452.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53608.log
2024-05-08T01:37:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58216.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59443.log
2024-05-08T01:37:57.074 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.44452.log.gz
2024-05-08T01:37:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.53608.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40565.log
2024-05-08T01:37:57.074 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58216.log.gz
2024-05-08T01:37:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.59443.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50630.log
2024-05-08T01:37:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.40565.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55354.log
2024-05-08T01:37:57.074 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53608.log.gz
2024-05-08T01:37:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.50630.log:	 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59443.log.gz
2024-05-08T01:37:57.074 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.40565.log.gz
2024-05-08T01:37:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61905.log
2024-05-08T01:37:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62832.log
2024-05-08T01:37:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.55354.log:	 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.50630.log.gz
2024-05-08T01:37:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.61905.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51433.log
2024-05-08T01:37:57.075 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55354.log.gz
2024-05-08T01:37:57.075 INFO:teuthology.orchestra.run.smithi142.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61905.log.gz
2024-05-08T01:37:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.62832.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65839.log
2024-05-08T01:37:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37873.log/var/log/ceph/cluster2-client.admin.51433.log:
2024-05-08T01:37:57.075 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62832.log.gz
2024-05-08T01:37:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.65839.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46043.log
2024-05-08T01:37:57.075 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.51433.log.gz
2024-05-08T01:37:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72316.log
2024-05-08T01:37:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.37873.log:	/var/log/ceph/cluster2-client.mirror.46043.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59590.log
2024-05-08T01:37:57.075 INFO:teuthology.orchestra.run.smithi142.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65839.log.gz
2024-05-08T01:37:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.72316.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39473.log
2024-05-08T01:37:57.075 INFO:teuthology.orchestra.run.smithi142.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.37873.log.gz/var/log/ceph/cluster1-client.mirror.59590.log:
2024-05-08T01:37:57.076 INFO:teuthology.orchestra.run.smithi142.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72316.log.gz
2024-05-08T01:37:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70817.log
2024-05-08T01:37:57.076 INFO:teuthology.orchestra.run.smithi142.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59590.log.gz
2024-05-08T01:37:57.076 INFO:teuthology.orchestra.run.smithi142.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46043.log.gz
2024-05-08T01:37:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.39473.log:	 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39473.log.gz
2024-05-08T01:37:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63478.log
2024-05-08T01:37:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.70817.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36327.log
2024-05-08T01:37:57.076 INFO:teuthology.orchestra.run.smithi142.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70817.log.gz
2024-05-08T01:37:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.63478.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36616.log
2024-05-08T01:37:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63478.log.gz
2024-05-08T01:37:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.36327.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74599.log
2024-05-08T01:37:57.076 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.36327.log.gz/var/log/ceph/cluster2-client.admin.36616.log:
2024-05-08T01:37:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69775.log
2024-05-08T01:37:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31281.log
2024-05-08T01:37:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.74599.log:	 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.36616.log.gz
2024-05-08T01:37:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74599.log.gz
2024-05-08T01:37:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.69775.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60741.log
2024-05-08T01:37:57.077 INFO:teuthology.orchestra.run.smithi142.stderr: 82.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69775.log.gz
2024-05-08T01:37:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.31281.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38421.log
2024-05-08T01:37:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:gzip/var/log/ceph/cluster1-client.admin.60741.log:	 -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64181.log
2024-05-08T01:37:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.38421.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53128.log
2024-05-08T01:37:57.077 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.60741.log.gz
2024-05-08T01:37:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.64181.log:	 92.3%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.66684.log.gz -5
2024-05-08T01:37:57.077 INFO:teuthology.orchestra.run.smithi142.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.35063.log
2024-05-08T01:37:57.077 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.38421.log.gz
2024-05-08T01:37:57.077 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30798.log
2024-05-08T01:37:57.077 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.64181.log.gz
2024-05-08T01:37:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.53128.log:	/var/log/ceph/cluster2-client.mirror.35063.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69591.log
2024-05-08T01:37:57.078 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35063.log.gz
2024-05-08T01:37:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.30798.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41920.log
2024-05-08T01:37:57.078 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53128.log.gz
2024-05-08T01:37:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.69591.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52588.log
2024-05-08T01:37:57.078 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.30798.log.gz
2024-05-08T01:37:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.41920.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52822.log
2024-05-08T01:37:57.078 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69591.log.gz
2024-05-08T01:37:57.078 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41920.log.gz
2024-05-08T01:37:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.52588.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47974.log
2024-05-08T01:37:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34668.log
2024-05-08T01:37:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.52822.log:	 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.52588.log.gz
2024-05-08T01:37:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.47974.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43176.log
2024-05-08T01:37:57.078 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.52822.log.gz
2024-05-08T01:37:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.34668.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48033.log
2024-05-08T01:37:57.079 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.34668.log.gz
2024-05-08T01:37:57.079 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.47974.log.gz
2024-05-08T01:37:57.079 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.43176.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67602.log
2024-05-08T01:37:57.079 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.48033.log:	 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.43176.log.gz
2024-05-08T01:37:57.079 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39836.log
2024-05-08T01:37:57.079 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36387.log
2024-05-08T01:37:57.079 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.67602.log:	 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48033.log.gz
2024-05-08T01:37:57.079 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.39836.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33591.log
2024-05-08T01:37:57.079 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67602.log.gz
2024-05-08T01:37:57.079 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.36387.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31886.log
2024-05-08T01:37:57.079 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.39836.log.gz
2024-05-08T01:37:57.079 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.33591.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44314.log
2024-05-08T01:37:57.079 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.36387.log.gz
2024-05-08T01:37:57.080 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.31886.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51373.log
2024-05-08T01:37:57.080 INFO:teuthology.orchestra.run.smithi142.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.admin.33591.log.gz
2024-05-08T01:37:57.080 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.44314.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71916.log
2024-05-08T01:37:57.080 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.admin.31886.log.gz
2024-05-08T01:37:57.080 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.51373.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37638.log
2024-05-08T01:37:57.080 INFO:teuthology.orchestra.run.smithi142.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44314.log.gz
2024-05-08T01:37:57.080 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.71916.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51797.log
2024-05-08T01:37:57.080 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.51373.log.gz
2024-05-08T01:37:57.080 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.37638.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39599.log
2024-05-08T01:37:57.080 INFO:teuthology.orchestra.run.smithi142.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31281.log.gz
2024-05-08T01:37:57.080 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.51797.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64146.log
2024-05-08T01:37:57.080 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.39599.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59965.log
2024-05-08T01:37:57.080 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.64146.log:	gzip 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.71916.log.gz -5
2024-05-08T01:37:57.080 INFO:teuthology.orchestra.run.smithi142.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.56419.log
2024-05-08T01:37:57.081 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.39599.log.gz
2024-05-08T01:37:57.081 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51797.log.gz
2024-05-08T01:37:57.081 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.59965.log:	 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64146.log.gz
2024-05-08T01:37:57.081 INFO:teuthology.orchestra.run.smithi142.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.37638.log.gz
2024-05-08T01:37:57.081 INFO:teuthology.orchestra.run.smithi142.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59965.log.gz
2024-05-08T01:37:57.081 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61308.log
2024-05-08T01:37:57.081 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.56419.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48204.log
2024-05-08T01:37:57.081 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.56419.log.gz
2024-05-08T01:37:57.081 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.61308.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72887.log
2024-05-08T01:37:57.081 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68249.log
2024-05-08T01:37:57.081 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.48204.log:	/var/log/ceph/cluster1-client.admin.72887.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41108.log
2024-05-08T01:37:57.081 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.48204.log.gz
2024-05-08T01:37:57.081 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.68249.log:	 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.72887.log.gz
2024-05-08T01:37:57.081 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55939.log
2024-05-08T01:37:57.082 INFO:teuthology.orchestra.run.smithi142.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68249.log.gz
2024-05-08T01:37:57.082 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.41108.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68173.log
2024-05-08T01:37:57.082 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.55939.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.41108.log.gz
2024-05-08T01:37:57.082 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66608.log
2024-05-08T01:37:57.082 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.55939.log.gz
2024-05-08T01:37:57.082 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.68173.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44660.log
2024-05-08T01:37:57.082 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63605.log
2024-05-08T01:37:57.082 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.66608.log:	 55.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66608.log.gz
2024-05-08T01:37:57.082 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.44660.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41984.log
2024-05-08T01:37:57.082 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.63605.log:	 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.44660.log.gz
2024-05-08T01:37:57.082 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36929.log
2024-05-08T01:37:57.082 INFO:teuthology.orchestra.run.smithi142.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63605.log.gz
2024-05-08T01:37:57.082 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.41984.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33408.log
2024-05-08T01:37:57.082 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.36929.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.31369.log
2024-05-08T01:37:57.083 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.41984.log.gz
2024-05-08T01:37:57.083 INFO:teuthology.orchestra.run.smithi142.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68173.log.gz
2024-05-08T01:37:57.083 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.33408.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34768.log
2024-05-08T01:37:57.083 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28809.log
2024-05-08T01:37:57.083 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.2.31369.log:	/var/log/ceph/cluster1-client.mirror.34768.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42017.log
2024-05-08T01:37:57.083 INFO:teuthology.orchestra.run.smithi142.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.33408.log.gz
2024-05-08T01:37:57.083 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.34768.log.gz
2024-05-08T01:37:57.083 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.28809.log:	  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28809.log.gz
2024-05-08T01:37:57.083 INFO:teuthology.orchestra.run.smithi142.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36929.log.gz
2024-05-08T01:37:57.083 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68202.log
2024-05-08T01:37:57.083 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71573.log
2024-05-08T01:37:57.083 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.42017.log:	/var/log/ceph/cluster2-client.mirror.68202.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32901.log
2024-05-08T01:37:57.083 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.42017.log.gz
2024-05-08T01:37:57.083 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.71573.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36083.log
2024-05-08T01:37:57.084 INFO:teuthology.orchestra.run.smithi142.stderr: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.71573.log.gz
2024-05-08T01:37:57.084 INFO:teuthology.orchestra.run.smithi142.stderr: 89.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56387.log
2024-05-08T01:37:57.084 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.68202.log.gz
2024-05-08T01:37:57.084 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.32901.log:	/var/log/ceph/cluster2-client.admin.36083.log:	 90.6% -- replaced with /var/log/ceph/cluster1-client.admin.32901.log.gz
2024-05-08T01:37:57.084 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.36083.log.gz
2024-05-08T01:37:57.084 INFO:teuthology.orchestra.run.smithi142.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61308.log.gz
2024-05-08T01:37:57.084 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66816.log
2024-05-08T01:37:57.084 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.56387.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65817.log
2024-05-08T01:37:57.084 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.66816.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32232.log
2024-05-08T01:37:57.084 INFO:teuthology.orchestra.run.smithi142.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66816.log.gz
2024-05-08T01:37:57.084 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.56387.log.gz
2024-05-08T01:37:57.084 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.65817.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63567.log
2024-05-08T01:37:57.084 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.32232.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44109.log
2024-05-08T01:37:57.084 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65817.log.gz
2024-05-08T01:37:57.085 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.63567.log:	 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.32232.log.gz
2024-05-08T01:37:57.085 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62668.log
2024-05-08T01:37:57.085 INFO:teuthology.orchestra.run.smithi142.stderr: 55.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63567.log.gz
2024-05-08T01:37:57.085 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31148.log
2024-05-08T01:37:57.085 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.44109.log:	/var/log/ceph/cluster1-client.mirror.62668.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.31313.log
2024-05-08T01:37:57.085 INFO:teuthology.orchestra.run.smithi142.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62668.log.gz
2024-05-08T01:37:57.085 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.31148.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60927.log
2024-05-08T01:37:57.085 INFO:teuthology.orchestra.run.smithi142.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44109.log.gz
2024-05-08T01:37:57.085 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51577.log
2024-05-08T01:37:57.085 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.0.31313.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.31148.log.gz
2024-05-08T01:37:57.085 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.60927.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50563.log
2024-05-08T01:37:57.085 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.51577.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67393.log
2024-05-08T01:37:57.085 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6%/var/log/ceph/cluster2-client.mirror.50563.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.51577.log.gz
2024-05-08T01:37:57.085 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29453.log
2024-05-08T01:37:57.086 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50563.log.gz
2024-05-08T01:37:57.086 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.67393.log:	 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67393.log.gz
2024-05-08T01:37:57.086 INFO:teuthology.orchestra.run.smithi142.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60927.log.gz
2024-05-08T01:37:57.086 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55453.log
2024-05-08T01:37:57.086 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.29453.log:	  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29453.log.gz
2024-05-08T01:37:57.086 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54698.log
2024-05-08T01:37:57.086 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70908.log
2024-05-08T01:37:57.086 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.55453.log:	/var/log/ceph/cluster1-client.admin.54698.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.24281.log
2024-05-08T01:37:57.086 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.55453.log.gz
2024-05-08T01:37:57.086 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.70908.log:	 88.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.54698.log.gz
2024-05-08T01:37:57.086 INFO:teuthology.orchestra.run.smithi142.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43753.log
2024-05-08T01:37:57.086 INFO:teuthology.orchestra.run.smithi142.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70908.log.gz
2024-05-08T01:37:57.086 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37269.log
2024-05-08T01:37:57.087 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.24281.log:	  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.24281.log.gz
2024-05-08T01:37:57.087 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.43753.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38296.log
2024-05-08T01:37:57.087 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.37269.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73488.log
2024-05-08T01:37:57.087 INFO:teuthology.orchestra.run.smithi142.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43753.log.gz
2024-05-08T01:37:57.087 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.38296.log:	 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.37269.log.gz
2024-05-08T01:37:57.087 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30040.log
2024-05-08T01:37:57.087 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29103.log
2024-05-08T01:37:57.087 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.38296.log.gz
2024-05-08T01:37:57.087 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.73488.log:	/var/log/ceph/cluster2-client.admin.30040.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36894.log
2024-05-08T01:37:57.087 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.73488.log.gz
2024-05-08T01:37:57.087 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.30040.log.gz
2024-05-08T01:37:57.087 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.29103.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41167.log
2024-05-08T01:37:57.087 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29103.log.gz
2024-05-08T01:37:57.087 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.36894.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68373.log
2024-05-08T01:37:57.088 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.41167.log:	 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.36894.log.gz
2024-05-08T01:37:57.088 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49253.log
2024-05-08T01:37:57.088 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.68373.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69652.log
2024-05-08T01:37:57.088 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.49253.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60829.log
2024-05-08T01:37:57.088 INFO:teuthology.orchestra.run.smithi142.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41167.log.gz
2024-05-08T01:37:57.088 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.69652.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.49253.log.gz
2024-05-08T01:37:57.088 INFO:teuthology.orchestra.run.smithi142.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68373.log.gz
2024-05-08T01:37:57.088 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72178.log
2024-05-08T01:37:57.088 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.60829.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66387.log
2024-05-08T01:37:57.088 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.72178.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52617.log
2024-05-08T01:37:57.088 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3%/var/log/ceph/cluster2-client.mirror.66387.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.72178.log.gz
2024-05-08T01:37:57.088 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59933.log
2024-05-08T01:37:57.088 INFO:teuthology.orchestra.run.smithi142.stderr: 93.0%/var/log/ceph/cluster1-client.admin.52617.log:	 -- replaced with /var/log/ceph/cluster1-client.mirror.60829.log.gz
2024-05-08T01:37:57.089 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64088.log
2024-05-08T01:37:57.089 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.52617.log.gz
2024-05-08T01:37:57.089 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.59933.log:	 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66387.log.gz
2024-05-08T01:37:57.089 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42484.log
2024-05-08T01:37:57.089 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46537.log
2024-05-08T01:37:57.089 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.64088.log:	/var/log/ceph/cluster2-client.admin.42484.log:	 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64088.log.gz
2024-05-08T01:37:57.089 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.42484.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47945.log
2024-05-08T01:37:57.089 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:57.089 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.46537.log:	 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59933.log.gz
2024-05-08T01:37:57.089 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.46537.log.gz
2024-05-08T01:37:57.089 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48587.log
2024-05-08T01:37:57.089 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.47945.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61215.log
2024-05-08T01:37:57.089 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.48587.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62216.log
2024-05-08T01:37:57.089 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.47945.log.gz
2024-05-08T01:37:57.090 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48587.log.gz
2024-05-08T01:37:57.090 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38090.log
2024-05-08T01:37:57.090 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.61215.log:	/var/log/ceph/cluster2-client.mirror.62216.log:	 83.0% -- replaced with /var/log/ceph/cluster2-client.mirror.61215.log.gz
2024-05-08T01:37:57.090 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58521.log
2024-05-08T01:37:57.090 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.38090.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64420.log
2024-05-08T01:37:57.090 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6%/var/log/ceph/cluster2-client.mirror.58521.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.38090.log.gz
2024-05-08T01:37:57.090 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58521.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48861.log
2024-05-08T01:37:57.090 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:57.090 INFO:teuthology.orchestra.run.smithi142.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62216.log.gz
2024-05-08T01:37:57.090 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.64420.log:	  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64420.log.gz
2024-05-08T01:37:57.090 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51275.log
2024-05-08T01:37:57.090 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.48861.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46328.log
2024-05-08T01:37:57.090 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.51275.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70727.log
2024-05-08T01:37:57.090 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.48861.log.gz
2024-05-08T01:37:57.091 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.46328.log:	 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.51275.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59799.log
2024-05-08T01:37:57.091 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:57.091 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.70727.log:	 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46328.log.gz
2024-05-08T01:37:57.091 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71858.log
2024-05-08T01:37:57.091 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70727.log.gz
2024-05-08T01:37:57.091 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.59799.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40833.log
2024-05-08T01:37:57.091 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.30331.log
2024-05-08T01:37:57.091 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.59799.log.gz
2024-05-08T01:37:57.091 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.71858.log:	/var/log/ceph/cluster2-client.admin.40833.log:	 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71858.log.gz
2024-05-08T01:37:57.091 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70520.log
2024-05-08T01:37:57.091 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.40833.log.gz
2024-05-08T01:37:57.091 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.2.30331.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66158.log
2024-05-08T01:37:57.091 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58194.log
2024-05-08T01:37:57.091 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.70520.log:	/var/log/ceph/cluster1-client.mirror.66158.log:	 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70520.log.gz
2024-05-08T01:37:57.092 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66158.log.gz
2024-05-08T01:37:57.092 INFO:teuthology.orchestra.run.smithi142.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.2.31369.log.gz
2024-05-08T01:37:57.092 INFO:teuthology.orchestra.run.smithi142.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.2.30331.log.gz
2024-05-08T01:37:57.092 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47125.log
2024-05-08T01:37:57.092 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58194.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51522.log
2024-05-08T01:37:57.092 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.47125.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43051.log
2024-05-08T01:37:57.092 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58194.log.gz
2024-05-08T01:37:57.092 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.51522.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43560.log
2024-05-08T01:37:57.092 INFO:teuthology.orchestra.run.smithi142.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47125.log.gz
2024-05-08T01:37:57.092 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.43051.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33315.log
2024-05-08T01:37:57.092 INFO:teuthology.orchestra.run.smithi142.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.51522.log.gz
2024-05-08T01:37:57.092 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.43560.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.43051.log.gz
2024-05-08T01:37:57.092 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45157.log
2024-05-08T01:37:57.092 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49751.log
2024-05-08T01:37:57.092 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.33315.log:	/var/log/ceph/cluster2-client.admin.45157.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33066.log
2024-05-08T01:37:57.093 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43560.log.gz
2024-05-08T01:37:57.093 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.33315.log.gz
2024-05-08T01:37:57.093 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.45157.log.gz
2024-05-08T01:37:57.093 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.49751.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.49751.log.gz
2024-05-08T01:37:57.093 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69249.log
2024-05-08T01:37:57.093 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.33066.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62367.log
2024-05-08T01:37:57.093 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.69249.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65906.log
2024-05-08T01:37:57.093 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.62367.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61409.log
2024-05-08T01:37:57.093 INFO:teuthology.orchestra.run.smithi142.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.33066.log.gz
2024-05-08T01:37:57.093 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69249.log.gz
2024-05-08T01:37:57.093 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.65906.log:	 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62367.log.gz
2024-05-08T01:37:57.093 INFO:teuthology.orchestra.run.smithi142.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65906.log.gz
2024-05-08T01:37:57.093 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47428.log
2024-05-08T01:37:57.094 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.61409.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65950.log
2024-05-08T01:37:57.094 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.47428.log:	 88.1%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.61409.log.gz
2024-05-08T01:37:57.094 INFO:teuthology.orchestra.run.smithi142.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.55035.log
2024-05-08T01:37:57.094 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.65950.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50182.log
2024-05-08T01:37:57.094 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.47428.log.gz
2024-05-08T01:37:57.094 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.55035.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42355.log
2024-05-08T01:37:57.094 INFO:teuthology.orchestra.run.smithi142.stderr:gzip 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65950.log.gz -5
2024-05-08T01:37:57.094 INFO:teuthology.orchestra.run.smithi142.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.28887.log
2024-05-08T01:37:57.094 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.55035.log.gz
2024-05-08T01:37:57.094 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.50182.log:	/var/log/ceph/cluster1-client.mirror.42355.log:	 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69652.log.gz
2024-05-08T01:37:57.094 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53388.log
2024-05-08T01:37:57.094 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42355.log.gz
2024-05-08T01:37:57.094 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.50182.log.gz
2024-05-08T01:37:57.094 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.28887.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72936.log
2024-05-08T01:37:57.095 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28887.log.gz
2024-05-08T01:37:57.095 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.53388.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47488.log
2024-05-08T01:37:57.095 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53388.log.gz
2024-05-08T01:37:57.095 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.72936.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30411.log
2024-05-08T01:37:57.095 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56600.log
2024-05-08T01:37:57.095 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.47488.log:	/var/log/ceph/cluster2-client.mirror.30411.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47284.log
2024-05-08T01:37:57.095 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.30411.log.gz
2024-05-08T01:37:57.095 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.47488.log.gz
2024-05-08T01:37:57.095 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.56600.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61616.log
2024-05-08T01:37:57.095 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.47284.log:	gzip -5 88.0% --verbose -- -- replaced with /var/log/ceph/cluster1-client.admin.56600.log.gz /var/log/ceph/cluster2-client.admin.52555.log
2024-05-08T01:37:57.095 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:57.095 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.61616.log:	gzip -5 --verbose -- 87.9% /var/log/ceph/cluster2-client.admin.49313.log
2024-05-08T01:37:57.095 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.47284.log.gz
2024-05-08T01:37:57.095 INFO:teuthology.orchestra.run.smithi142.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72936.log.gz
2024-05-08T01:37:57.096 INFO:teuthology.orchestra.run.smithi142.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61616.log.gz
2024-05-08T01:37:57.096 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.52555.log:	 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.52555.log.gz
2024-05-08T01:37:57.096 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70839.log
2024-05-08T01:37:57.096 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.49313.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71730.log
2024-05-08T01:37:57.096 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.49313.log.gz
2024-05-08T01:37:57.096 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.70839.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30331.log
2024-05-08T01:37:57.096 INFO:teuthology.orchestra.run.smithi142.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70839.log.gz
2024-05-08T01:37:57.096 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69025.log
2024-05-08T01:37:57.096 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.71730.log:	/var/log/ceph/cluster2-client.mirror.30331.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68715.log
2024-05-08T01:37:57.096 INFO:teuthology.orchestra.run.smithi142.stderr: 86.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71730.log.gz
2024-05-08T01:37:57.096 INFO:teuthology.orchestra.run.smithi142.stderr: 81.1% -- replaced with /var/log/ceph/cluster2-client.mirror.30331.log.gz
2024-05-08T01:37:57.096 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.69025.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59251.log
2024-05-08T01:37:57.096 INFO:teuthology.orchestra.run.smithi142.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69025.log.gz
2024-05-08T01:37:57.096 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.68715.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59656.log
2024-05-08T01:37:57.097 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.59251.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39980.log
2024-05-08T01:37:57.097 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.68715.log.gz
2024-05-08T01:37:57.097 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.59656.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56978.log
2024-05-08T01:37:57.097 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.39980.log:	gzip 89.3% -5 -- replaced with /var/log/ceph/cluster2-client.mirror.59251.log.gz --verbose
2024-05-08T01:37:57.097 INFO:teuthology.orchestra.run.smithi142.stderr: -- /var/log/ceph/cluster2-client.mirror.30211.log
2024-05-08T01:37:57.097 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59656.log.gz
2024-05-08T01:37:57.097 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.56978.log:	 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39980.log.gz
2024-05-08T01:37:57.097 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57130.log
2024-05-08T01:37:57.097 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.30211.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73579.log
2024-05-08T01:37:57.097 INFO:teuthology.orchestra.run.smithi142.stderr: 58.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30211.log.gz
2024-05-08T01:37:57.097 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56978.log.gz
2024-05-08T01:37:57.097 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.57130.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67730.log
2024-05-08T01:37:57.097 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.73579.log:	 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.57130.log.gz
2024-05-08T01:37:57.097 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58851.log
2024-05-08T01:37:57.098 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.73579.log.gz
2024-05-08T01:37:57.098 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.67730.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55868.log
2024-05-08T01:37:57.098 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.58851.log:	 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67730.log.gz
2024-05-08T01:37:57.098 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54291.log
2024-05-08T01:37:57.098 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.55868.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31855.log
2024-05-08T01:37:57.098 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4%/var/log/ceph/cluster1-client.admin.54291.log:	 -- replaced with /var/log/ceph/cluster1-client.mirror.55868.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47066.log
2024-05-08T01:37:57.098 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:57.098 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.31855.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64037.log
2024-05-08T01:37:57.098 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.54291.log.gz
2024-05-08T01:37:57.098 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41563.log
2024-05-08T01:37:57.098 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.47066.log:	/var/log/ceph/cluster2-client.mirror.64037.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29846.log
2024-05-08T01:37:57.098 INFO:teuthology.orchestra.run.smithi142.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.admin.31855.log.gz
2024-05-08T01:37:57.098 INFO:teuthology.orchestra.run.smithi142.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58851.log.gz
2024-05-08T01:37:57.098 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.41563.log:	 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64037.log.gz
2024-05-08T01:37:57.099 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.47066.log.gz
2024-05-08T01:37:57.099 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.41563.log.gz
2024-05-08T01:37:57.099 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43368.log
2024-05-08T01:37:57.099 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.29846.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55299.log
2024-05-08T01:37:57.099 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29846.log.gz
2024-05-08T01:37:57.099 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.43368.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37490.log
2024-05-08T01:37:57.099 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43368.log.gz
2024-05-08T01:37:57.099 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.55299.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71596.log
2024-05-08T01:37:57.099 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.37490.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53935.log
2024-05-08T01:37:57.099 INFO:teuthology.orchestra.run.smithi142.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.55299.log.gz
2024-05-08T01:37:57.099 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.71596.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70983.log
2024-05-08T01:37:57.099 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.37490.log.gz
2024-05-08T01:37:57.099 INFO:teuthology.orchestra.run.smithi142.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71596.log.gz
2024-05-08T01:37:57.100 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.53935.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71232.log
2024-05-08T01:37:57.100 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53549.log
2024-05-08T01:37:57.100 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.70983.log:	 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53935.log.gz
2024-05-08T01:37:57.100 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.71232.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47852.log
2024-05-08T01:37:57.100 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70983.log.gz
2024-05-08T01:37:57.100 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.53549.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38980.log
2024-05-08T01:37:57.100 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.47852.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39528.log
2024-05-08T01:37:57.100 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.53549.log.gz
2024-05-08T01:37:57.100 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.38980.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69456.log
2024-05-08T01:37:57.100 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33898.log
2024-05-08T01:37:57.100 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.38980.log.gz/var/log/ceph/cluster1-client.mirror.39528.log:
2024-05-08T01:37:57.100 INFO:teuthology.orchestra.run.smithi142.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47852.log.gz
2024-05-08T01:37:57.100 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.69456.log:	 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39528.log.gz
2024-05-08T01:37:57.100 INFO:teuthology.orchestra.run.smithi142.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71232.log.gz
2024-05-08T01:37:57.100 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66203.log
2024-05-08T01:37:57.101 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74125.log
2024-05-08T01:37:57.101 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.33898.log:	/var/log/ceph/cluster1-client.mirror.66203.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44286.log
2024-05-08T01:37:57.101 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1%/var/log/ceph/cluster2-client.mirror.74125.log:	 -- replaced with /var/log/ceph/cluster1-client.mirror.66203.log.gz
2024-05-08T01:37:57.101 INFO:teuthology.orchestra.run.smithi142.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.admin.33898.log.gz
2024-05-08T01:37:57.101 INFO:teuthology.orchestra.run.smithi142.stderr:gzip  0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74125.log.gz -5
2024-05-08T01:37:57.101 INFO:teuthology.orchestra.run.smithi142.stderr: --verbose -- 91.4% /var/log/ceph/cluster1-client.admin.33836.log
2024-05-08T01:37:57.101 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.69456.log.gz
2024-05-08T01:37:57.101 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.44286.log:	 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.44286.log.gz
2024-05-08T01:37:57.101 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42722.log
2024-05-08T01:37:57.101 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.33836.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68023.log
2024-05-08T01:37:57.101 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.42722.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73517.log
2024-05-08T01:37:57.101 INFO:teuthology.orchestra.run.smithi142.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.33836.log.gz
2024-05-08T01:37:57.101 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.68023.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45188.log
2024-05-08T01:37:57.102 INFO:teuthology.orchestra.run.smithi142.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42722.log.gz
2024-05-08T01:37:57.102 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34718.log
2024-05-08T01:37:57.102 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.73517.log:	 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.68023.log.gz
2024-05-08T01:37:57.102 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.45188.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70150.log
2024-05-08T01:37:57.102 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73517.log.gz
2024-05-08T01:37:57.102 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.34718.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54937.log
2024-05-08T01:37:57.102 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.34718.log.gz 87.9%
2024-05-08T01:37:57.102 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.45188.log.gz
2024-05-08T01:37:57.102 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.70150.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36949.log
2024-05-08T01:37:57.102 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.54937.log:	gzip -5 --verbose 87.6% -- /var/log/ceph/cluster1-client.mirror.70885.log -- replaced with /var/log/ceph/cluster1-client.mirror.70150.log.gz
2024-05-08T01:37:57.102 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:57.102 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.36949.log:	 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54937.log.gz
2024-05-08T01:37:57.102 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43501.log
2024-05-08T01:37:57.102 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.70885.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48139.log
2024-05-08T01:37:57.103 INFO:teuthology.orchestra.run.smithi142.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70885.log.gz
2024-05-08T01:37:57.103 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.43501.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74247.log
2024-05-08T01:37:57.103 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.48139.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58973.log
2024-05-08T01:37:57.103 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.43501.log.gz
2024-05-08T01:37:57.103 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.74247.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66082.log
2024-05-08T01:37:57.103 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48139.log.gz
2024-05-08T01:37:57.103 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74247.log.gz
2024-05-08T01:37:57.103 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59198.log
2024-05-08T01:37:57.103 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58973.log:	 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.36949.log.gz
2024-05-08T01:37:57.103 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.66082.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44629.log
2024-05-08T01:37:57.103 INFO:teuthology.orchestra.run.smithi142.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58973.log.gz
2024-05-08T01:37:57.103 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.59198.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59731.log
2024-05-08T01:37:57.103 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.44629.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51145.log
2024-05-08T01:37:57.103 INFO:teuthology.orchestra.run.smithi142.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66082.log.gz
2024-05-08T01:37:57.103 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.59731.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40079.log
2024-05-08T01:37:57.104 INFO:teuthology.orchestra.run.smithi142.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59198.log.gz
2024-05-08T01:37:57.104 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.44629.log.gz
2024-05-08T01:37:57.104 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0%/var/log/ceph/cluster1-client.mirror.51145.log:	 -- replaced with /var/log/ceph/cluster1-client.mirror.59731.log.gz
2024-05-08T01:37:57.104 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51145.log.gz
2024-05-08T01:37:57.104 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67301.log
2024-05-08T01:37:57.104 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.40079.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54517.log
2024-05-08T01:37:57.104 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.67301.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61018.log
2024-05-08T01:37:57.104 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.40079.log.gz
2024-05-08T01:37:57.104 INFO:teuthology.orchestra.run.smithi142.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67301.log.gz
2024-05-08T01:37:57.104 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.54517.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44195.log
2024-05-08T01:37:57.104 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45468.log
2024-05-08T01:37:57.104 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.61018.log:	 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.54517.log.gz
2024-05-08T01:37:57.104 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.44195.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39445.log
2024-05-08T01:37:57.104 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.45468.log:	 88.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61957.log
2024-05-08T01:37:57.104 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.44195.log.gz
2024-05-08T01:37:57.105 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.39445.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62088.log
2024-05-08T01:37:57.105 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.45468.log.gz
2024-05-08T01:37:57.105 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47699.log
2024-05-08T01:37:57.105 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.39445.log.gz
2024-05-08T01:37:57.105 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.61957.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48485.log
2024-05-08T01:37:57.105 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.62088.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29938.log
2024-05-08T01:37:57.105 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.47699.log:	 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61957.log.gz
2024-05-08T01:37:57.105 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.48485.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52762.log
2024-05-08T01:37:57.105 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.47699.log.gz 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62088.log.gz
2024-05-08T01:37:57.105 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:57.105 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.29938.log:	 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.48485.log.gz
2024-05-08T01:37:57.105 INFO:teuthology.orchestra.run.smithi142.stderr:  4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.29938.log.gz
2024-05-08T01:37:57.105 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74781.log
2024-05-08T01:37:57.105 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.52762.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59913.log
2024-05-08T01:37:57.106 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.74781.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.52762.log.gz
2024-05-08T01:37:57.106 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58794.log
2024-05-08T01:37:57.106 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74781.log.gz
2024-05-08T01:37:57.106 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.59913.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71053.log
2024-05-08T01:37:57.106 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.58794.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56265.log
2024-05-08T01:37:57.106 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59913.log.gz
2024-05-08T01:37:57.106 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.71053.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35274.log
2024-05-08T01:37:57.106 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68902.log
2024-05-08T01:37:57.106 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71053.log.gz
2024-05-08T01:37:57.106 INFO:teuthology.orchestra.run.smithi142.stderr: 90.9%/var/log/ceph/cluster2-client.mirror.56265.log:	 -- replaced with /var/log/ceph/cluster1-client.admin.58794.log.gz
2024-05-08T01:37:57.106 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.35274.log:	 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.35274.log.gz
2024-05-08T01:37:57.106 INFO:teuthology.orchestra.run.smithi142.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56265.log.gz
2024-05-08T01:37:57.106 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61061.log
2024-05-08T01:37:57.106 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65884.log
2024-05-08T01:37:57.107 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.68902.log:	/var/log/ceph/cluster2-client.mirror.61061.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47764.log
2024-05-08T01:37:57.107 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0%/var/log/ceph/cluster2-client.mirror.65884.log:	 -- replaced with /var/log/ceph/cluster1-client.mirror.68902.log.gz
2024-05-08T01:37:57.107 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55181.log
2024-05-08T01:37:57.107 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49076.log
2024-05-08T01:37:57.107 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61061.log.gz
2024-05-08T01:37:57.107 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.47764.log:	/var/log/ceph/cluster1-client.admin.55181.log:	 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65884.log.gz
2024-05-08T01:37:57.107 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.47764.log.gz
2024-05-08T01:37:57.107 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.55181.log.gz
2024-05-08T01:37:57.107 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66303.log
2024-05-08T01:37:57.107 INFO:teuthology.orchestra.run.smithi142.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61018.log.gz
2024-05-08T01:37:57.107 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.31374.log
2024-05-08T01:37:57.107 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.49076.log:	/var/log/ceph/cluster2-client.mirror.66303.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74807.log
2024-05-08T01:37:57.107 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.49076.log.gz
2024-05-08T01:37:57.107 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.3.31374.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72339.log
2024-05-08T01:37:57.108 INFO:teuthology.orchestra.run.smithi142.stderr: 19.9% -- replaced with /var/log/ceph/cluster1-client.mirror.3.31374.log.gz
2024-05-08T01:37:57.108 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.74807.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29017.log
2024-05-08T01:37:57.108 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74807.log.gz
2024-05-08T01:37:57.108 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.72339.log:	 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66303.log.gz
2024-05-08T01:37:57.108 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68844.log
2024-05-08T01:37:57.108 INFO:teuthology.orchestra.run.smithi142.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.72339.log.gz
2024-05-08T01:37:57.108 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.29017.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42515.log
2024-05-08T01:37:57.108 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29017.log.gz
2024-05-08T01:37:57.108 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28861.log
2024-05-08T01:37:57.108 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.68844.log:	/var/log/ceph/cluster2-client.admin.42515.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45437.log
2024-05-08T01:37:57.108 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.42515.log.gz
2024-05-08T01:37:57.108 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.28861.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32839.log
2024-05-08T01:37:57.108 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28861.log.gz
2024-05-08T01:37:57.108 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.45437.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61279.log
2024-05-08T01:37:57.109 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.32839.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32485.log
2024-05-08T01:37:57.109 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.45437.log.gz
2024-05-08T01:37:57.109 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.61279.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41079.log
2024-05-08T01:37:57.109 INFO:teuthology.orchestra.run.smithi142.stderr: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68844.log.gz
2024-05-08T01:37:57.109 INFO:teuthology.orchestra.run.smithi142.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.admin.32839.log.gz/var/log/ceph/cluster1-client.admin.32485.log:
2024-05-08T01:37:57.109 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36647.log
2024-05-08T01:37:57.109 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59118.log
2024-05-08T01:37:57.109 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61279.log.gz
2024-05-08T01:37:57.109 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.41079.log:	 90.6%/var/log/ceph/cluster2-client.admin.36647.log:	 -- replaced with /var/log/ceph/cluster1-client.admin.32485.log.gz
2024-05-08T01:37:57.109 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72235.log
2024-05-08T01:37:57.109 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.41079.log.gz
2024-05-08T01:37:57.109 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.59118.log:	 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.36647.log.gz
2024-05-08T01:37:57.109 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36730.log
2024-05-08T01:37:57.109 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.72235.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38055.log
2024-05-08T01:37:57.109 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.36730.log:	 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72235.log.gz
2024-05-08T01:37:57.110 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36116.log
2024-05-08T01:37:57.110 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36730.log.gz
2024-05-08T01:37:57.110 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.38055.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62111.log
2024-05-08T01:37:57.110 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.36116.log:	 87.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49589.log
2024-05-08T01:37:57.110 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.38055.log.gz
2024-05-08T01:37:57.110 INFO:teuthology.orchestra.run.smithi142.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59118.log.gz
2024-05-08T01:37:57.110 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.62111.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34044.log
2024-05-08T01:37:57.110 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.36116.log.gz
2024-05-08T01:37:57.110 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53274.log
2024-05-08T01:37:57.110 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.49589.log:	/var/log/ceph/cluster1-client.admin.34044.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56691.log
2024-05-08T01:37:57.110 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62111.log.gz
2024-05-08T01:37:57.110 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.49589.log.gz
2024-05-08T01:37:57.110 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.53274.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60125.log
2024-05-08T01:37:57.110 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.34044.log.gz
2024-05-08T01:37:57.111 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.69657.log
2024-05-08T01:37:57.111 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.56691.log:	 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.53274.log.gz
2024-05-08T01:37:57.111 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.60125.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58584.log
2024-05-08T01:37:57.111 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9%/var/log/ceph/cluster1-client.mirror.1.69657.log:	 -- replaced with /var/log/ceph/cluster2-client.admin.56691.log.gz
2024-05-08T01:37:57.111 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68419.log
2024-05-08T01:37:57.111 INFO:teuthology.orchestra.run.smithi142.stderr: 19.9% -- replaced with /var/log/ceph/cluster1-client.mirror.1.69657.log.gz
2024-05-08T01:37:57.111 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41711.log
2024-05-08T01:37:57.111 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.58584.log:	/var/log/ceph/cluster2-client.mirror.68419.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68781.log
2024-05-08T01:37:57.111 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.58584.log.gz
2024-05-08T01:37:57.111 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4%/var/log/ceph/cluster1-client.admin.41711.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.68419.log.gz
2024-05-08T01:37:57.111 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41223.log
2024-05-08T01:37:57.111 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.41711.log.gz
2024-05-08T01:37:57.111 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.68781.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40270.log
2024-05-08T01:37:57.112 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.41223.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42543.log
2024-05-08T01:37:57.112 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.40270.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57583.log
2024-05-08T01:37:57.112 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.41223.log.gz
2024-05-08T01:37:57.112 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.42543.log:	 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.40270.log.gz
2024-05-08T01:37:57.112 INFO:teuthology.orchestra.run.smithi142.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68781.log.gz
2024-05-08T01:37:57.112 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42543.log.gz
2024-05-08T01:37:57.112 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38511.log
2024-05-08T01:37:57.112 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.57583.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37781.log
2024-05-08T01:37:57.112 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.38511.log:	 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57583.log.gz
2024-05-08T01:37:57.112 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49960.log
2024-05-08T01:37:57.112 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.37781.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50843.log
2024-05-08T01:37:57.112 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.38511.log.gz
2024-05-08T01:37:57.112 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.49960.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72565.log
2024-05-08T01:37:57.112 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.37781.log.gz
2024-05-08T01:37:57.112 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49960.log.gz
2024-05-08T01:37:57.113 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.50843.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72101.log
2024-05-08T01:37:57.113 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.72565.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52238.log
2024-05-08T01:37:57.113 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.50843.log.gz
2024-05-08T01:37:57.113 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.72101.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53455.log
2024-05-08T01:37:57.113 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.52238.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57896.log
2024-05-08T01:37:57.113 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72101.log.gz
2024-05-08T01:37:57.113 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.53455.log:	 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.53455.log.gz
2024-05-08T01:37:57.113 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52238.log.gz
2024-05-08T01:37:57.113 INFO:teuthology.orchestra.run.smithi142.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.72565.log.gz
2024-05-08T01:37:57.113 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46632.log
2024-05-08T01:37:57.113 INFO:teuthology.orchestra.run.smithi142.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60125.log.gz
2024-05-08T01:37:57.113 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.57896.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69160.log
2024-05-08T01:37:57.113 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.46632.log:	 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.57896.log.gz
2024-05-08T01:37:57.113 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30160.log
2024-05-08T01:37:57.113 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45616.log
2024-05-08T01:37:57.114 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.69160.log:	 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.46632.log.gz
2024-05-08T01:37:57.114 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.30160.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.30336.log
2024-05-08T01:37:57.114 INFO:teuthology.orchestra.run.smithi142.stderr:  4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.30160.log.gz
2024-05-08T01:37:57.114 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.45616.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.73584.log
2024-05-08T01:37:57.114 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.3.30336.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57225.log
2024-05-08T01:37:57.114 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.45616.log.gz
2024-05-08T01:37:57.114 INFO:teuthology.orchestra.run.smithi142.stderr: 19.9% -- replaced with /var/log/ceph/cluster1-client.mirror.3.30336.log.gz
2024-05-08T01:37:57.114 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.1.73584.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71369.log
2024-05-08T01:37:57.114 INFO:teuthology.orchestra.run.smithi142.stderr: 20.5% -- replaced with /var/log/ceph/cluster2-client.mirror.1.73584.log.gz
2024-05-08T01:37:57.114 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.57225.log:	 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69160.log.gz
2024-05-08T01:37:57.114 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37929.log
2024-05-08T01:37:57.114 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5%/var/log/ceph/cluster1-client.admin.71369.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.57225.log.gz
2024-05-08T01:37:57.114 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48623.log
2024-05-08T01:37:57.114 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.37929.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30958.log
2024-05-08T01:37:57.115 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.71369.log.gz
2024-05-08T01:37:57.115 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.48623.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47097.log
2024-05-08T01:37:57.115 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.30958.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37057.log
2024-05-08T01:37:57.115 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.48623.log.gz
2024-05-08T01:37:57.115 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.47097.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30126.log
2024-05-08T01:37:57.115 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.30958.log.gz
2024-05-08T01:37:57.115 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55240.log
2024-05-08T01:37:57.115 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.37057.log:	 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.47097.log.gz
2024-05-08T01:37:57.115 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.30126.log:	 69.8% -- replaced with /var/log/ceph/cluster2-client.admin.30126.log.gz
2024-05-08T01:37:57.115 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.37057.log.gz
2024-05-08T01:37:57.115 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.69724.log
2024-05-08T01:37:57.115 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.55240.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52114.log
2024-05-08T01:37:57.115 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.2.69724.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41771.log
2024-05-08T01:37:57.115 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.55240.log.gz
2024-05-08T01:37:57.116 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.52114.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61115.log
2024-05-08T01:37:57.116 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.41771.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30177.log
2024-05-08T01:37:57.116 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.52114.log.gz
2024-05-08T01:37:57.116 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.61115.log:	 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.41771.log.gz
2024-05-08T01:37:57.116 INFO:teuthology.orchestra.run.smithi142.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61115.log.gz
2024-05-08T01:37:57.116 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49040.log
2024-05-08T01:37:57.116 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.30177.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.69680.log
2024-05-08T01:37:57.116 INFO:teuthology.orchestra.run.smithi142.stderr:  4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.30177.log.gz
2024-05-08T01:37:57.116 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.49040.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31369.log
2024-05-08T01:37:57.116 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.49040.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48104.log
2024-05-08T01:37:57.116 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:57.116 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.1.69680.log:	/var/log/ceph/cluster2-client.mirror.31369.log:	 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.31369.log.gz
2024-05-08T01:37:57.116 INFO:teuthology.orchestra.run.smithi142.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.2.69724.log.gz
2024-05-08T01:37:57.116 INFO:teuthology.orchestra.run.smithi142.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37929.log.gz
2024-05-08T01:37:57.117 INFO:teuthology.orchestra.run.smithi142.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.69680.log.gz
2024-05-08T01:37:57.117 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60515.log
2024-05-08T01:37:57.117 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.48104.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60573.log
2024-05-08T01:37:57.117 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48104.log.gz
2024-05-08T01:37:57.117 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.60515.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36176.log
2024-05-08T01:37:57.117 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.60573.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33181.log
2024-05-08T01:37:57.117 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60515.log.gz
2024-05-08T01:37:57.117 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.36176.log:	 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60573.log.gz
2024-05-08T01:37:57.117 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46293.log
2024-05-08T01:37:57.117 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.33181.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61593.log
2024-05-08T01:37:57.117 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.36176.log.gz
2024-05-08T01:37:57.118 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.46293.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45765.log
2024-05-08T01:37:57.118 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46293.log.gz
2024-05-08T01:37:57.118 INFO:teuthology.orchestra.run.smithi142.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.admin.33181.log.gz
2024-05-08T01:37:57.118 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.61593.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60387.log
2024-05-08T01:37:57.118 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68648.log
2024-05-08T01:37:57.118 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.45765.log:	/var/log/ceph/cluster2-client.mirror.60387.log:	gzip -5 --verbose -- 87.5% /var/log/ceph/cluster1-client.mirror.49005.log -- replaced with /var/log/ceph/cluster2-client.mirror.45765.log.gz
2024-05-08T01:37:57.118 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:57.118 INFO:teuthology.orchestra.run.smithi142.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.68648.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.60387.log.gz
2024-05-08T01:37:57.118 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42105.log
2024-05-08T01:37:57.118 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68648.log.gz
2024-05-08T01:37:57.118 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.49005.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59148.log
2024-05-08T01:37:57.118 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.49005.log.gz
2024-05-08T01:37:57.118 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55903.log
2024-05-08T01:37:57.118 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.59148.log:	/var/log/ceph/cluster2-client.mirror.42105.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73257.log
2024-05-08T01:37:57.119 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.55903.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53487.log
2024-05-08T01:37:57.119 INFO:teuthology.orchestra.run.smithi142.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59148.log.gz
2024-05-08T01:37:57.119 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.73257.log:	 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55903.log.gz
2024-05-08T01:37:57.119 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42105.log.gz
2024-05-08T01:37:57.119 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.73257.log.gz
2024-05-08T01:37:57.119 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67208.log
2024-05-08T01:37:57.119 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.53487.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67433.log
2024-05-08T01:37:57.119 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.67208.log:	 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.53487.log.gz
2024-05-08T01:37:57.119 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54969.log
2024-05-08T01:37:57.119 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.67433.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70281.log
2024-05-08T01:37:57.119 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.54969.log:	 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67208.log.gz
2024-05-08T01:37:57.119 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53580.log
2024-05-08T01:37:57.119 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67433.log.gz
2024-05-08T01:37:57.119 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.70281.log:	 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.54969.log.gz
2024-05-08T01:37:57.119 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58283.log
2024-05-08T01:37:57.120 INFO:teuthology.orchestra.run.smithi142.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70281.log.gz
2024-05-08T01:37:57.120 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.53580.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70780.log
2024-05-08T01:37:57.120 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58283.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31976.log
2024-05-08T01:37:57.120 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.53580.log.gz
2024-05-08T01:37:57.120 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.70780.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73711.log
2024-05-08T01:37:57.120 INFO:teuthology.orchestra.run.smithi142.stderr: 89.2%/var/log/ceph/cluster1-client.mirror.31976.log:	 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70780.log.gz
2024-05-08T01:37:57.120 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.58283.log.gz
2024-05-08T01:37:57.120 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73537.log
2024-05-08T01:37:57.120 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.31346.log
2024-05-08T01:37:57.120 INFO:teuthology.orchestra.run.smithi142.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.61593.log.gz
2024-05-08T01:37:57.120 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.73711.log:	/var/log/ceph/cluster2-client.mirror.73537.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61236.log
2024-05-08T01:37:57.120 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7%/var/log/ceph/cluster1-client.mirror.2.31346.log:	 -- replaced with /var/log/ceph/cluster1-client.mirror.73711.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28784.log
2024-05-08T01:37:57.120 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:57.120 INFO:teuthology.orchestra.run.smithi142.stderr: 19.9% -- replaced with /var/log/ceph/cluster1-client.mirror.2.31346.log.gz
2024-05-08T01:37:57.121 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.61236.log:	 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.31976.log.gz
2024-05-08T01:37:57.121 INFO:teuthology.orchestra.run.smithi142.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73537.log.gz
2024-05-08T01:37:57.121 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71460.log
2024-05-08T01:37:57.121 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.28784.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62274.log
2024-05-08T01:37:57.121 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28784.log.gz
2024-05-08T01:37:57.121 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.71460.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67865.log
2024-05-08T01:37:57.121 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.62274.log:	 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.71460.log.gz
2024-05-08T01:37:57.121 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55709.log
2024-05-08T01:37:57.121 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.67865.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34323.log
2024-05-08T01:37:57.121 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.55709.log:	 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62274.log.gz
2024-05-08T01:37:57.121 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67865.log.gz
2024-05-08T01:37:57.121 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44017.log
2024-05-08T01:37:57.121 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.34323.log:	 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.55709.log.gz
2024-05-08T01:37:57.121 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.34323.log.gz
2024-05-08T01:37:57.121 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60907.log
2024-05-08T01:37:57.122 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.44017.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29528.log
2024-05-08T01:37:57.205 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.60907.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48362.log
2024-05-08T01:37:57.205 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.44017.log.gz
2024-05-08T01:37:57.205 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.29528.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48393.log
2024-05-08T01:37:57.205 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29528.log.gz
2024-05-08T01:37:57.205 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.48362.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49721.log
2024-05-08T01:37:57.205 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.48393.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.73579.log
2024-05-08T01:37:57.205 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.48362.log.gz
2024-05-08T01:37:57.205 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.49721.log:	 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60907.log.gz
2024-05-08T01:37:57.205 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.48393.log.gz
2024-05-08T01:37:57.205 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.49721.log.gz
2024-05-08T01:37:57.205 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32808.log
2024-05-08T01:37:57.205 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.0.73579.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63584.log
2024-05-08T01:37:57.205 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.32808.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53787.log
2024-05-08T01:37:57.206 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.63584.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63461.log
2024-05-08T01:37:57.206 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.32808.log.gz
2024-05-08T01:37:57.206 INFO:teuthology.orchestra.run.smithi142.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63584.log.gz/var/log/ceph/cluster1-client.mirror.53787.log:
2024-05-08T01:37:57.206 INFO:teuthology.orchestra.run.smithi142.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.61236.log.gz
2024-05-08T01:37:57.206 INFO:teuthology.orchestra.run.smithi142.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53787.log.gz
2024-05-08T01:37:57.206 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49782.log
2024-05-08T01:37:57.206 INFO:teuthology.orchestra.run.smithi142.stderr: 91.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.73579.log.gz
2024-05-08T01:37:57.206 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.63461.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68397.log
2024-05-08T01:37:57.206 INFO:teuthology.orchestra.run.smithi142.stderr: 63.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63461.log.gz
2024-05-08T01:37:57.206 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.49782.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55486.log
2024-05-08T01:37:57.206 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.68397.log:	 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.49782.log.gz
2024-05-08T01:37:57.206 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29353.log
2024-05-08T01:37:57.206 INFO:teuthology.orchestra.run.smithi142.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68397.log.gz
2024-05-08T01:37:57.207 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.admin.55486.log:	 -- /var/log/ceph/cluster1-client.mirror.1.30303.log
2024-05-08T01:37:57.207 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.29353.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57035.log
2024-05-08T01:37:57.207 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29353.log.gz
2024-05-08T01:37:57.207 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.55486.log.gz
2024-05-08T01:37:57.207 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.1.30303.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68127.log
2024-05-08T01:37:57.207 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.57035.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65695.log
2024-05-08T01:37:57.207 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.57035.log.gz
2024-05-08T01:37:57.207 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.68127.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40172.log
2024-05-08T01:37:57.207 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.65695.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51769.log
2024-05-08T01:37:57.207 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.40172.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72619.log
2024-05-08T01:37:57.207 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4%/var/log/ceph/cluster1-client.admin.51769.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.65695.log.gz
2024-05-08T01:37:57.207 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.40172.log.gz
2024-05-08T01:37:57.207 INFO:teuthology.orchestra.run.smithi142.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68127.log.gz
2024-05-08T01:37:57.207 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.51769.log.gz
2024-05-08T01:37:57.208 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55389.log
2024-05-08T01:37:57.208 INFO:teuthology.orchestra.run.smithi142.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.30303.log.gz
2024-05-08T01:37:57.208 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.72619.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58262.log
2024-05-08T01:37:57.208 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72619.log.gz
2024-05-08T01:37:57.208 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.55389.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.24219.log
2024-05-08T01:37:57.208 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71620.log
2024-05-08T01:37:57.208 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58262.log:	 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55389.log.gz
2024-05-08T01:37:57.208 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.24219.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56001.log
2024-05-08T01:37:57.208 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.24219.log.gz
2024-05-08T01:37:57.208 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.71620.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74199.log
2024-05-08T01:37:57.208 INFO:teuthology.orchestra.run.smithi142.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58262.log.gz 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71620.log.gz
2024-05-08T01:37:57.208 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:57.208 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.56001.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32454.log
2024-05-08T01:37:57.208 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49492.log
2024-05-08T01:37:57.208 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.74199.log:	 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.56001.log.gz
2024-05-08T01:37:57.209 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74199.log.gz
2024-05-08T01:37:57.209 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.32454.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45922.log
2024-05-08T01:37:57.209 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.49492.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.32454.log.gz
2024-05-08T01:37:57.209 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48716.log
2024-05-08T01:37:57.209 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37232.log
2024-05-08T01:37:57.209 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49492.log.gz
2024-05-08T01:37:57.209 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.45922.log:	/var/log/ceph/cluster1-client.admin.48716.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28437.log
2024-05-08T01:37:57.209 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.45922.log.gz
2024-05-08T01:37:57.209 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.37232.log:	 87.7%gzip -- replaced with /var/log/ceph/cluster1-client.admin.48716.log.gz -5
2024-05-08T01:37:57.209 INFO:teuthology.orchestra.run.smithi142.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.50053.log
2024-05-08T01:37:57.209 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.28437.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56919.log
2024-05-08T01:37:57.209 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28437.log.gz
2024-05-08T01:37:57.209 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.50053.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38451.log
2024-05-08T01:37:57.209 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.56919.log:	 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50053.log.gz
2024-05-08T01:37:57.210 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29095.log
2024-05-08T01:37:57.210 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.38451.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54642.log
2024-05-08T01:37:57.210 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.56919.log.gz
2024-05-08T01:37:57.210 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.29095.log:	  0.0%gzip -- replaced with /var/log/ceph/cluster2-client.admin.29095.log.gz
2024-05-08T01:37:57.210 INFO:teuthology.orchestra.run.smithi142.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.51404.log
2024-05-08T01:37:57.210 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.38451.log.gz
2024-05-08T01:37:57.210 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35230.log
2024-05-08T01:37:57.210 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.54642.log:	/var/log/ceph/cluster1-client.admin.51404.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47459.log
2024-05-08T01:37:57.210 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.35230.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37027.log
2024-05-08T01:37:57.210 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.51404.log.gz
2024-05-08T01:37:57.210 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.54642.log.gz
2024-05-08T01:37:57.210 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.47459.log:	 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.35230.log.gz
2024-05-08T01:37:57.210 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.47459.log.gz
2024-05-08T01:37:57.211 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51857.log
2024-05-08T01:37:57.211 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.37027.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53842.log
2024-05-08T01:37:57.211 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.37027.log.gz
2024-05-08T01:37:57.211 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.51857.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58831.log
2024-05-08T01:37:57.211 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.53842.log:	 -- /var/log/ceph/cluster2-client.mirror.61840.log
2024-05-08T01:37:57.211 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.51857.log.gz
2024-05-08T01:37:57.211 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58831.log:	 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53842.log.gz
2024-05-08T01:37:57.211 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56825.log
2024-05-08T01:37:57.211 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.61840.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60878.log
2024-05-08T01:37:57.211 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.56825.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.73711.log
2024-05-08T01:37:57.211 INFO:teuthology.orchestra.run.smithi142.stderr: 89.2% 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56825.log.gz
2024-05-08T01:37:57.211 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.61840.log.gz/var/log/ceph/cluster1-client.mirror.60878.log:
2024-05-08T01:37:57.211 INFO:teuthology.orchestra.run.smithi142.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.58831.log.gz
2024-05-08T01:37:57.212 INFO:teuthology.orchestra.run.smithi142.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60878.log.gz
2024-05-08T01:37:57.212 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60666.log
2024-05-08T01:37:57.212 INFO:teuthology.orchestra.run.smithi142.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37232.log.gz
2024-05-08T01:37:57.212 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.3.73711.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48457.log
2024-05-08T01:37:57.212 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.60666.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57260.log
2024-05-08T01:37:57.212 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.48457.log:	 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60666.log.gz
2024-05-08T01:37:57.212 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71210.log
2024-05-08T01:37:57.212 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.57260.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38770.log
2024-05-08T01:37:57.212 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.48457.log.gz
2024-05-08T01:37:57.212 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57260.log.gz
2024-05-08T01:37:57.212 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.71210.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67788.log
2024-05-08T01:37:57.212 INFO:teuthology.orchestra.run.smithi142.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.3.73711.log.gz
2024-05-08T01:37:57.212 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.38770.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34818.log
2024-05-08T01:37:57.212 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.67788.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59418.log
2024-05-08T01:37:57.213 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.38770.log.gz
2024-05-08T01:37:57.213 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67788.log.gz
2024-05-08T01:37:57.213 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.34818.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30275.log
2024-05-08T01:37:57.213 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34818.log.gz
2024-05-08T01:37:57.213 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.59418.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70376.log
2024-05-08T01:37:57.213 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.30275.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51110.log
2024-05-08T01:37:57.213 INFO:teuthology.orchestra.run.smithi142.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59418.log.gz 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71210.log.gz
2024-05-08T01:37:57.213 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.70376.log:
2024-05-08T01:37:57.213 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42599.log
2024-05-08T01:37:57.213 INFO:teuthology.orchestra.run.smithi142.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70376.log.gz
2024-05-08T01:37:57.213 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.51110.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50326.log
2024-05-08T01:37:57.213 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51110.log.gz
2024-05-08T01:37:57.213 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.42599.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67547.log
2024-05-08T01:37:57.213 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.50326.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74100.log
2024-05-08T01:37:57.213 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.42599.log.gz
2024-05-08T01:37:57.213 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.67547.log:	 88.1%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33646.log
2024-05-08T01:37:57.214 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.50326.log.gz
2024-05-08T01:37:57.214 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.74100.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43439.log
2024-05-08T01:37:57.214 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74100.log.gz
2024-05-08T01:37:57.214 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.33646.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61755.log
2024-05-08T01:37:57.214 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.43439.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67191.log
2024-05-08T01:37:57.214 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.33646.log.gz
2024-05-08T01:37:57.214 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.61755.log:	 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.43439.log.gz
2024-05-08T01:37:57.214 INFO:teuthology.orchestra.run.smithi142.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61755.log.gz
2024-05-08T01:37:57.214 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66281.log
2024-05-08T01:37:57.214 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.67191.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73101.log
2024-05-08T01:37:57.214 INFO:teuthology.orchestra.run.smithi142.stderr: 57.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67191.log.gz
2024-05-08T01:37:57.214 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.66281.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61354.log
2024-05-08T01:37:57.214 INFO:teuthology.orchestra.run.smithi142.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30275.log.gz
2024-05-08T01:37:57.214 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43604.log
2024-05-08T01:37:57.215 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.73101.log:	/var/log/ceph/cluster1-client.mirror.61354.log:	 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66281.log.gz
2024-05-08T01:37:57.215 INFO:teuthology.orchestra.run.smithi142.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61354.log.gz
2024-05-08T01:37:57.215 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.73101.log.gz
2024-05-08T01:37:57.215 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72990.log
2024-05-08T01:37:57.215 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.43604.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74547.log
2024-05-08T01:37:57.215 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.72990.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32547.log
2024-05-08T01:37:57.215 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.43604.log.gz
2024-05-08T01:37:57.215 INFO:teuthology.orchestra.run.smithi142.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.admin.72990.log.gz
2024-05-08T01:37:57.215 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.74547.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47251.log
2024-05-08T01:37:57.215 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74547.log.gz
2024-05-08T01:37:57.215 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.32547.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41498.log
2024-05-08T01:37:57.215 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.47251.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41952.log
2024-05-08T01:37:57.215 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.41498.log:	 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.32547.log.gz
2024-05-08T01:37:57.215 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41376.log
2024-05-08T01:37:57.216 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.47251.log.gz
2024-05-08T01:37:57.216 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62463.log
2024-05-08T01:37:57.216 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41498.log.gz
2024-05-08T01:37:57.216 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.41952.log:	/var/log/ceph/cluster1-client.mirror.41376.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45343.log
2024-05-08T01:37:57.216 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.41952.log.gz
2024-05-08T01:37:57.216 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55801.log
2024-05-08T01:37:57.216 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9%/var/log/ceph/cluster1-client.mirror.62463.log:	 -- replaced with /var/log/ceph/cluster1-client.mirror.41376.log.gz
2024-05-08T01:37:57.216 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.45343.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58053.log
2024-05-08T01:37:57.216 INFO:teuthology.orchestra.run.smithi142.stderr: 93.9% 86.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67547.log.gz
2024-05-08T01:37:57.216 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.62463.log.gz
2024-05-08T01:37:57.216 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.55801.log:	 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45343.log.gz
2024-05-08T01:37:57.216 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64505.log
2024-05-08T01:37:57.216 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58053.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69616.log
2024-05-08T01:37:57.216 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55801.log.gz
2024-05-08T01:37:57.216 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.64505.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30109.log
2024-05-08T01:37:57.217 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.69616.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47215.log
2024-05-08T01:37:57.217 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.30109.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71189.log
2024-05-08T01:37:57.217 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69616.log.gz
2024-05-08T01:37:57.217 INFO:teuthology.orchestra.run.smithi142.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.admin.30109.log.gz
2024-05-08T01:37:57.217 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.47215.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65673.log
2024-05-08T01:37:57.217 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47215.log.gz/var/log/ceph/cluster2-client.mirror.71189.log:
2024-05-08T01:37:57.217 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68877.log
2024-05-08T01:37:57.217 INFO:teuthology.orchestra.run.smithi142.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64505.log.gz
2024-05-08T01:37:57.217 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67232.log
2024-05-08T01:37:57.217 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.65673.log:	/var/log/ceph/cluster1-client.mirror.68877.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50151.log
2024-05-08T01:37:57.217 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71189.log.gz
2024-05-08T01:37:57.217 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.67232.log:	 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68877.log.gz
2024-05-08T01:37:57.217 INFO:teuthology.orchestra.run.smithi142.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67232.log.gz
2024-05-08T01:37:57.218 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65673.log.gz
2024-05-08T01:37:57.218 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44983.log
2024-05-08T01:37:57.218 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.50151.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37176.log
2024-05-08T01:37:57.218 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.44983.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63112.log
2024-05-08T01:37:57.218 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.50151.log.gz
2024-05-08T01:37:57.218 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.37176.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41592.log
2024-05-08T01:37:57.218 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.44983.log.gz
2024-05-08T01:37:57.218 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54554.log
2024-05-08T01:37:57.218 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.63112.log:	 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.37176.log.gz
2024-05-08T01:37:57.218 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.41592.log:	 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63112.log.gz
2024-05-08T01:37:57.218 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.41592.log.gz
2024-05-08T01:37:57.218 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65761.log
2024-05-08T01:37:57.218 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.54554.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45583.log
2024-05-08T01:37:57.219 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.65761.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41348.log
2024-05-08T01:37:57.219 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.54554.log.gz
2024-05-08T01:37:57.219 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.45583.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44050.log
2024-05-08T01:37:57.219 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.41348.log:	 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65761.log.gz
2024-05-08T01:37:57.219 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59168.log
2024-05-08T01:37:57.219 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.45583.log.gz
2024-05-08T01:37:57.219 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54145.log
2024-05-08T01:37:57.219 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.41348.log.gz
2024-05-08T01:37:57.219 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.44050.log:	/var/log/ceph/cluster1-client.mirror.59168.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29628.log
2024-05-08T01:37:57.219 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.54145.log:	gzip 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58053.log.gz -5
2024-05-08T01:37:57.219 INFO:teuthology.orchestra.run.smithi142.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.39179.log
2024-05-08T01:37:57.219 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.44050.log.gz
2024-05-08T01:37:57.219 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.29628.log:	  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29628.log.gz
2024-05-08T01:37:57.219 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.54145.log.gz
2024-05-08T01:37:57.219 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72271.log
2024-05-08T01:37:57.220 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.39179.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56890.log
2024-05-08T01:37:57.220 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.72271.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61530.log
2024-05-08T01:37:57.220 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.39179.log.gz
2024-05-08T01:37:57.220 INFO:teuthology.orchestra.run.smithi142.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.72271.log.gz
2024-05-08T01:37:57.220 INFO:teuthology.orchestra.run.smithi142.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59168.log.gz
2024-05-08T01:37:57.220 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.56890.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45552.log
2024-05-08T01:37:57.220 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73223.log
2024-05-08T01:37:57.220 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.61530.log:	 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.56890.log.gz
2024-05-08T01:37:57.220 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.45552.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60858.log
2024-05-08T01:37:57.220 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36418.log
2024-05-08T01:37:57.220 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.73223.log:	 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.45552.log.gzgzip -5 --verbose 89.6% -- /var/log/ceph/cluster1-client.mirror.2.66756.log -- replaced with /var/log/ceph/cluster2-client.mirror.61530.log.gz
2024-05-08T01:37:57.220 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:57.220 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:57.221 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.60858.log:	/var/log/ceph/cluster2-client.admin.36418.log:	 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73223.log.gz
2024-05-08T01:37:57.221 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.36418.log.gz
2024-05-08T01:37:57.221 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53333.log
2024-05-08T01:37:57.221 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.2.66756.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34437.log
2024-05-08T01:37:57.221 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.53333.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34864.log
2024-05-08T01:37:57.221 INFO:teuthology.orchestra.run.smithi142.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60858.log.gz/var/log/ceph/cluster2-client.mirror.34437.log:
2024-05-08T01:37:57.221 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37148.log
2024-05-08T01:37:57.221 INFO:teuthology.orchestra.run.smithi142.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.53333.log.gz
2024-05-08T01:37:57.221 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.34437.log.gz
2024-05-08T01:37:57.221 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.34864.log:	 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.34864.log.gz
2024-05-08T01:37:57.221 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63363.log
2024-05-08T01:37:57.221 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.37148.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53100.log
2024-05-08T01:37:57.221 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.63363.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39354.log
2024-05-08T01:37:57.221 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.37148.log.gz
2024-05-08T01:37:57.222 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.53100.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72008.log
2024-05-08T01:37:57.222 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40982.log
2024-05-08T01:37:57.222 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.53100.log.gz
2024-05-08T01:37:57.222 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.39354.log:	 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.66756.log.gz
2024-05-08T01:37:57.222 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.72008.log:	 87.3% 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72008.log.gz
2024-05-08T01:37:57.222 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.39354.log.gz
2024-05-08T01:37:57.222 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56857.log
2024-05-08T01:37:57.222 INFO:teuthology.orchestra.run.smithi142.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63363.log.gz
2024-05-08T01:37:57.222 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.40982.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44724.log
2024-05-08T01:37:57.222 INFO:teuthology.orchestra.run.smithi142.stderr: 87.6%/var/log/ceph/cluster1-client.admin.56857.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.40982.log.gz
2024-05-08T01:37:57.222 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53668.log
2024-05-08T01:37:57.222 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.44724.log:	 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.56857.log.gz
2024-05-08T01:37:57.222 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73998.log
2024-05-08T01:37:57.223 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.44724.log.gz
2024-05-08T01:37:57.223 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.53668.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35877.log
2024-05-08T01:37:57.223 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.73998.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34635.log
2024-05-08T01:37:57.223 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1%  0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73998.log.gz
2024-05-08T01:37:57.223 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.53668.log.gz
2024-05-08T01:37:57.223 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.35877.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66336.log
2024-05-08T01:37:57.223 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29503.log
2024-05-08T01:37:57.223 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.34635.log:	 86.6% -- replaced with /var/log/ceph/cluster2-client.mirror.35877.log.gz
2024-05-08T01:37:57.223 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.66336.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64249.log
2024-05-08T01:37:57.223 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.29503.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48744.log
2024-05-08T01:37:57.223 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29503.log.gz
2024-05-08T01:37:57.223 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.64249.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57841.log
2024-05-08T01:37:57.223 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.48744.log:	 87.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69919.log
2024-05-08T01:37:57.223 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.64249.log.gz
2024-05-08T01:37:57.223 INFO:teuthology.orchestra.run.smithi142.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.34635.log.gz
2024-05-08T01:37:57.224 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.57841.log:	 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66336.log.gz
2024-05-08T01:37:57.224 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.48744.log.gz
2024-05-08T01:37:57.224 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57841.log.gz
2024-05-08T01:37:57.224 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58001.log
2024-05-08T01:37:57.224 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.69919.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74149.log
2024-05-08T01:37:57.224 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.58001.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31018.log
2024-05-08T01:37:57.224 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69919.log.gz
2024-05-08T01:37:57.224 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.74149.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37381.log
2024-05-08T01:37:57.224 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74149.log.gz
2024-05-08T01:37:57.224 INFO:teuthology.orchestra.run.smithi142.stderr: 90.9% -- replaced with /var/log/ceph/cluster2-client.admin.58001.log.gz
2024-05-08T01:37:57.224 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.31018.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36211.log
2024-05-08T01:37:57.224 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.37381.log:	 88.8% -- replaced with /var/log/ceph/cluster2-client.admin.31018.log.gz
2024-05-08T01:37:57.224 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50415.log
2024-05-08T01:37:57.224 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.37381.log.gz
2024-05-08T01:37:57.225 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.36211.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log
2024-05-08T01:37:57.225 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.50415.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40112.log
2024-05-08T01:37:57.225 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-osd.0.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70862.log
2024-05-08T01:37:57.225 INFO:teuthology.orchestra.run.smithi142.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50415.log.gz
2024-05-08T01:37:57.225 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.40112.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33867.log
2024-05-08T01:37:57.225 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62003.log
2024-05-08T01:37:57.225 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.40112.log.gz
2024-05-08T01:37:57.225 INFO:teuthology.orchestra.run.smithi142.stderr: 91.9%/var/log/ceph/cluster1-client.mirror.70862.log:	/var/log/ceph/cluster1-client.admin.33867.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.36211.log.gz
2024-05-08T01:37:57.225 INFO:teuthology.orchestra.run.smithi142.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70862.log.gz
2024-05-08T01:37:57.225 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.33867.log.gz
2024-05-08T01:37:57.225 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45823.log
2024-05-08T01:37:57.225 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.62003.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58370.log
2024-05-08T01:37:57.225 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0%/var/log/ceph/cluster2-client.mirror.45823.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71158.log
2024-05-08T01:37:57.225 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.62003.log.gz
2024-05-08T01:37:57.226 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58370.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53759.log
2024-05-08T01:37:57.226 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45823.log.gz
2024-05-08T01:37:57.226 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.71158.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73369.log
2024-05-08T01:37:57.226 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58370.log.gz
2024-05-08T01:37:57.226 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.53759.log:	 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71158.log.gz
2024-05-08T01:37:57.226 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54204.log
2024-05-08T01:37:57.226 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.73369.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29904.log
2024-05-08T01:37:57.226 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.53759.log.gz
2024-05-08T01:37:57.226 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.54204.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68956.log
2024-05-08T01:37:57.226 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.73369.log.gz
2024-05-08T01:37:57.226 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28945.log
2024-05-08T01:37:57.226 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.29904.log:	/var/log/ceph/cluster1-client.mirror.68956.log:	 70.2% -- replaced with /var/log/ceph/cluster1-client.admin.29904.log.gz
2024-05-08T01:37:57.226 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54204.log.gz
2024-05-08T01:37:57.226 INFO:teuthology.orchestra.run.smithi142.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68956.log.gz
2024-05-08T01:37:57.227 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35334.log
2024-05-08T01:37:57.227 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.28945.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69680.log
2024-05-08T01:37:57.227 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28945.log.gz
2024-05-08T01:37:57.227 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.35334.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43665.log
2024-05-08T01:37:57.227 INFO:teuthology.orchestra.run.smithi142.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.35334.log.gz
2024-05-08T01:37:57.227 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.69680.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61798.log
2024-05-08T01:37:57.227 INFO:teuthology.orchestra.run.smithi142.stderr: 82.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69680.log.gz
2024-05-08T01:37:57.227 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.43665.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32516.log
2024-05-08T01:37:57.227 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.61798.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.35230.log
2024-05-08T01:37:57.227 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.43665.log.gz
2024-05-08T01:37:57.227 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.32516.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.35207.log
2024-05-08T01:37:57.227 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.1.35230.log:	 88.1% 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61798.log.gz
2024-05-08T01:37:57.227 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71338.log
2024-05-08T01:37:57.227 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.32516.log.gz
2024-05-08T01:37:57.228 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.1.35207.log:	 20.5% -- replaced with /var/log/ceph/cluster2-client.mirror.1.35207.log.gz
2024-05-08T01:37:57.228 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.66717.log
2024-05-08T01:37:57.228 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.71338.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54383.log
2024-05-08T01:37:57.228 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.2.66717.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58304.log
2024-05-08T01:37:57.228 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.71338.log.gz
2024-05-08T01:37:57.228 INFO:teuthology.orchestra.run.smithi142.stderr: 19.9% -- replaced with /var/log/ceph/cluster1-client.mirror.2.66717.log.gz
2024-05-08T01:37:57.228 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.54383.log:	 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.54383.log.gz
2024-05-08T01:37:57.228 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38916.log
2024-05-08T01:37:57.228 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.58304.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39661.log
2024-05-08T01:37:57.228 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.38916.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50783.log
2024-05-08T01:37:57.228 INFO:teuthology.orchestra.run.smithi142.stderr: 90.3%/var/log/ceph/cluster2-client.admin.39661.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.58304.log.gz
2024-05-08T01:37:57.228 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.38916.log.gz
2024-05-08T01:37:57.228 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.39661.log.gz
2024-05-08T01:37:57.229 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62530.log
2024-05-08T01:37:57.229 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.50783.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67170.log
2024-05-08T01:37:57.229 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.62530.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55972.log
2024-05-08T01:37:57.229 INFO:teuthology.orchestra.run.smithi142.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62530.log.gz
2024-05-08T01:37:57.229 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0%/var/log/ceph/cluster1-client.mirror.67170.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.50783.log.gz
2024-05-08T01:37:57.229 INFO:teuthology.orchestra.run.smithi142.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67170.log.gz
2024-05-08T01:37:57.229 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45496.log
2024-05-08T01:37:57.229 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67455.log
2024-05-08T01:37:57.229 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.55972.log:	gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.45496.log:	 /var/log/ceph/cluster2-client.admin.29099.log
2024-05-08T01:37:57.229 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.67455.log:	 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.55972.log.gz
2024-05-08T01:37:57.229 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45496.log.gz
2024-05-08T01:37:57.229 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67455.log.gz
2024-05-08T01:37:57.229 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61928.log
2024-05-08T01:37:57.229 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.29099.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.66712.log
2024-05-08T01:37:57.230 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29099.log.gz
2024-05-08T01:37:57.230 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.61928.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63013.log
2024-05-08T01:37:57.230 INFO:teuthology.orchestra.run.smithi142.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61928.log.gz
2024-05-08T01:37:57.230 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.1.66712.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36016.log
2024-05-08T01:37:57.230 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59098.log
2024-05-08T01:37:57.230 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.63013.log:	/var/log/ceph/cluster2-client.mirror.36016.log:	 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36016.log.gz
2024-05-08T01:37:57.230 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63013.log.gz
2024-05-08T01:37:57.230 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30600.log
2024-05-08T01:37:57.230 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30245.log
2024-05-08T01:37:57.230 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.59098.log:	/var/log/ceph/cluster2-client.mirror.30600.log:	 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59098.log.gz
2024-05-08T01:37:57.230 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41253.log
2024-05-08T01:37:57.230 INFO:teuthology.orchestra.run.smithi142.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30600.log.gz
2024-05-08T01:37:57.230 INFO:teuthology.orchestra.run.smithi142.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.1.66712.log.gz
2024-05-08T01:37:57.230 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.30245.log:	 55.6% -- replaced with /var/log/ceph/cluster2-client.mirror.30245.log.gz
2024-05-08T01:37:57.231 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49402.log
2024-05-08T01:37:57.231 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.41253.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41317.log
2024-05-08T01:37:57.231 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.49402.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52417.log
2024-05-08T01:37:57.231 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.41253.log.gz
2024-05-08T01:37:57.231 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57158.log
2024-05-08T01:37:57.231 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.41317.log:	 89.1%/var/log/ceph/cluster2-client.mirror.52417.log:	 -- replaced with /var/log/ceph/cluster2-client.mirror.49402.log.gz
2024-05-08T01:37:57.231 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.41317.log.gz
2024-05-08T01:37:57.231 INFO:teuthology.orchestra.run.smithi142.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52417.log.gz
2024-05-08T01:37:57.231 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66016.log
2024-05-08T01:37:57.231 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47793.log
2024-05-08T01:37:57.231 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.57158.log:	gzip/var/log/ceph/cluster2-client.mirror.66016.log:	 -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58647.log
2024-05-08T01:37:57.231 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.47793.log:	 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.57158.log.gz
2024-05-08T01:37:57.231 INFO:teuthology.orchestra.run.smithi142.stderr: 89.2% 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.47793.log.gz
2024-05-08T01:37:57.232 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.66016.log.gz
2024-05-08T01:37:57.232 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41046.log
2024-05-08T01:37:57.232 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74271.log
2024-05-08T01:37:57.232 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.58647.log:	/var/log/ceph/cluster2-client.admin.41046.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59778.log
2024-05-08T01:37:57.232 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.74271.log:	  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74271.log.gz 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.41046.log.gz
2024-05-08T01:37:57.232 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:57.232 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57968.log
2024-05-08T01:37:57.232 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.59778.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33004.log
2024-05-08T01:37:57.232 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.57968.log:	 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58647.log.gz
2024-05-08T01:37:57.232 INFO:teuthology.orchestra.run.smithi142.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59778.log.gz
2024-05-08T01:37:57.232 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.57968.log.gz
2024-05-08T01:37:57.232 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55546.log
2024-05-08T01:37:57.232 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72812.log
2024-05-08T01:37:57.232 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.33004.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70620.log
2024-05-08T01:37:57.233 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.55546.log:	/var/log/ceph/cluster2-client.mirror.72812.log:	 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.55546.log.gz
2024-05-08T01:37:57.233 INFO:teuthology.orchestra.run.smithi142.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.admin.33004.log.gz
2024-05-08T01:37:57.233 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72812.log.gz
2024-05-08T01:37:57.233 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49618.log
2024-05-08T01:37:57.233 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49374.log
2024-05-08T01:37:57.233 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.70620.log:	/var/log/ceph/cluster1-client.admin.49618.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74573.log
2024-05-08T01:37:57.233 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.49374.log:	 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70620.log.gz
2024-05-08T01:37:57.233 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.49618.log.gz
2024-05-08T01:37:57.233 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60701.log
2024-05-08T01:37:57.233 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.49374.log.gz
2024-05-08T01:37:57.233 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47731.log
2024-05-08T01:37:57.233 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.74573.log:	/var/log/ceph/cluster2-client.mirror.60701.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34937.log
2024-05-08T01:37:57.233 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74573.log.gz
2024-05-08T01:37:57.233 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.47731.log:	 88.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.69125.log
2024-05-08T01:37:57.234 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.60701.log.gz
2024-05-08T01:37:57.234 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.47731.log.gz
2024-05-08T01:37:57.234 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.34937.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36828.log
2024-05-08T01:37:57.234 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.69125.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51303.log
2024-05-08T01:37:57.234 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7%/var/log/ceph/cluster1-client.admin.36828.log:	 -- replaced with /var/log/ceph/cluster1-client.admin.69125.log.gz
2024-05-08T01:37:57.234 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.36828.log.gz
2024-05-08T01:37:57.234 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42422.log
2024-05-08T01:37:57.234 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55773.log
2024-05-08T01:37:57.234 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.51303.log:	/var/log/ceph/cluster1-client.admin.42422.log:	 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.42422.log.gz
2024-05-08T01:37:57.234 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51303.log.gz
2024-05-08T01:37:57.234 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38331.log
2024-05-08T01:37:57.234 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71551.log
2024-05-08T01:37:57.234 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.55773.log:	/var/log/ceph/cluster2-client.mirror.38331.log:	 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.55773.log.gz
2024-05-08T01:37:57.235 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57811.log
2024-05-08T01:37:57.235 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.71551.log:	 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71551.log.gz
2024-05-08T01:37:57.235 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43403.log
2024-05-08T01:37:57.235 INFO:teuthology.orchestra.run.smithi142.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38331.log.gz
2024-05-08T01:37:57.235 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.57811.log:	 90.6% -- replaced with /var/log/ceph/cluster2-client.admin.57811.log.gz
2024-05-08T01:37:57.235 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43917.log
2024-05-08T01:37:57.235 INFO:teuthology.orchestra.run.smithi142.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.34937.log.gz
2024-05-08T01:37:57.235 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37088.log
2024-05-08T01:37:57.235 INFO:teuthology.orchestra.run.smithi142.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.35230.log.gz
2024-05-08T01:37:57.235 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.43403.log:	/var/log/ceph/cluster2-client.mirror.43917.log:	 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43403.log.gz
2024-05-08T01:37:57.235 INFO:teuthology.orchestra.run.smithi142.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43917.log.gz
2024-05-08T01:37:57.235 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59613.log
2024-05-08T01:37:57.235 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.37088.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54176.log
2024-05-08T01:37:57.235 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.59613.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72066.log
2024-05-08T01:37:57.236 INFO:teuthology.orchestra.run.smithi142.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.37088.log.gz
2024-05-08T01:37:57.236 INFO:teuthology.orchestra.run.smithi142.stderr: 86.7%/var/log/ceph/cluster2-client.admin.54176.log:	 -- replaced with /var/log/ceph/cluster1-client.mirror.59613.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.73570.log
2024-05-08T01:37:57.236 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:37:57.236 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.72066.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34181.log
2024-05-08T01:37:57.236 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.54176.log.gz
2024-05-08T01:37:57.236 INFO:teuthology.orchestra.run.smithi142.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72066.log.gz
2024-05-08T01:37:57.236 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40658.log
2024-05-08T01:37:57.236 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.0.73570.log:	/var/log/ceph/cluster2-client.mirror.34181.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62132.log
2024-05-08T01:37:57.236 INFO:teuthology.orchestra.run.smithi142.stderr: 20.5% -- replaced with /var/log/ceph/cluster2-client.mirror.0.73570.log.gz
2024-05-08T01:37:57.236 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.40658.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log
2024-05-08T01:37:57.236 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.62132.log:	 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34181.log.gz
2024-05-08T01:37:57.236 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74729.log
2024-05-08T01:37:57.236 INFO:teuthology.orchestra.run.smithi142.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.40658.log.gz
2024-05-08T01:37:57.236 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52793.log
2024-05-08T01:37:57.237 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-osd.2.log:	/var/log/ceph/cluster2-client.admin.74729.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47518.log
2024-05-08T01:37:57.237 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62132.log.gz
2024-05-08T01:37:57.237 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.74729.log.gz
2024-05-08T01:37:57.237 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.52793.log:	 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.52793.log.gz
2024-05-08T01:37:57.237 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67414.log
2024-05-08T01:37:57.237 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.47518.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.66689.log
2024-05-08T01:37:57.237 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.67414.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66125.log
2024-05-08T01:37:57.237 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.47518.log.gz
2024-05-08T01:37:57.237 INFO:teuthology.orchestra.run.smithi142.stderr: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67414.log.gz
2024-05-08T01:37:57.237 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.1.66689.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37346.log
2024-05-08T01:37:57.237 INFO:teuthology.orchestra.run.smithi142.stderr: 19.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.66689.log.gz
2024-05-08T01:37:57.237 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.66125.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54759.log
2024-05-08T01:37:57.237 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2%/var/log/ceph/cluster1-client.mirror.37346.log:	 -- replaced with /var/log/ceph/cluster1-client.mirror.66125.log.gz
2024-05-08T01:37:57.237 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49677.log
2024-05-08T01:37:57.237 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.37346.log.gz
2024-05-08T01:37:57.238 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61861.log
2024-05-08T01:37:57.238 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.54759.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39866.log
2024-05-08T01:37:57.238 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.49677.log:	 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.54759.log.gz
2024-05-08T01:37:57.238 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.61861.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59530.log
2024-05-08T01:37:57.238 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.admin.39866.log:	 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.49677.log.gz
2024-05-08T01:37:57.238 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61861.log.gz
2024-05-08T01:37:57.238 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.39866.log.gz
2024-05-08T01:37:57.238 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72747.log
2024-05-08T01:37:57.238 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59072.log
2024-05-08T01:37:57.238 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.59530.log:	/var/log/ceph/cluster2-client.mirror.72747.log:	gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66712.log
2024-05-08T01:37:57.238 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.mirror.59072.log:	 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72747.log.gz
2024-05-08T01:37:57.238 INFO:teuthology.orchestra.run.smithi142.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59530.log.gz
2024-05-08T01:37:57.238 INFO:teuthology.orchestra.run.smithi142.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59072.log.gz
2024-05-08T01:37:57.238 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38234.log
2024-05-08T01:37:57.239 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster2-client.mirror.66712.log:	gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62508.log
2024-05-08T01:37:57.239 INFO:teuthology.orchestra.run.smithi142.stderr: 81.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66712.log.gz
2024-05-08T01:37:57.239 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cluster1-client.admin.38234.log:	/var/log/ceph/cluster1-client.mirror.62508.log:	 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.38234.log.gz
2024-05-08T01:37:57.239 INFO:teuthology.orchestra.run.smithi142.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62508.log.gz
2024-05-08T01:37:57.241 INFO:teuthology.orchestra.run.smithi142.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.31313.log.gz
2024-05-08T01:37:57.403 INFO:teuthology.orchestra.run.smithi106.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz
2024-05-08T01:37:57.523 INFO:teuthology.orchestra.run.smithi142.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz
2024-05-08T01:38:00.253 INFO:teuthology.orchestra.run.smithi106.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz
2024-05-08T01:38:00.396 INFO:teuthology.orchestra.run.smithi106.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz
2024-05-08T01:38:00.658 INFO:teuthology.orchestra.run.smithi142.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz
2024-05-08T01:38:01.372 INFO:teuthology.orchestra.run.smithi142.stderr: 94.5% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz
2024-05-08T01:38:01.437 INFO:teuthology.orchestra.run.smithi106.stderr: 94.6% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz
2024-05-08T01:38:01.439 INFO:teuthology.orchestra.run.smithi106.stderr:
2024-05-08T01:38:01.439 INFO:teuthology.orchestra.run.smithi106.stderr:real	0m4.909s
2024-05-08T01:38:01.439 INFO:teuthology.orchestra.run.smithi106.stderr:user	0m13.241s
2024-05-08T01:38:01.439 INFO:teuthology.orchestra.run.smithi106.stderr:sys	0m0.469s
2024-05-08T01:38:01.853 INFO:teuthology.orchestra.run.smithi142.stderr: 94.6% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz
2024-05-08T01:38:01.854 INFO:teuthology.orchestra.run.smithi142.stderr:
2024-05-08T01:38:01.854 INFO:teuthology.orchestra.run.smithi142.stderr:real	0m5.315s
2024-05-08T01:38:01.854 INFO:teuthology.orchestra.run.smithi142.stderr:user	0m15.791s
2024-05-08T01:38:01.854 INFO:teuthology.orchestra.run.smithi142.stderr:sys	0m1.008s
2024-05-08T01:38:01.855 INFO:tasks.ceph:Archiving logs...
2024-05-08T01:38:01.855 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/log/ceph to /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696665/remote/smithi106/log
2024-05-08T01:38:01.858 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/log/ceph -- .
2024-05-08T01:38:03.460 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/log/ceph to /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696665/remote/smithi142/log
2024-05-08T01:38:03.460 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /var/log/ceph -- .
2024-05-08T01:38:06.257 DEBUG:teuthology.run_tasks:Unwinding manager install
2024-05-08T01:38:06.273 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer...
2024-05-08T01:38:06.274 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer
2024-05-08T01:38:06.276 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer
2024-05-08T01:38:06.335 DEBUG:teuthology.run_tasks:Unwinding manager clock
2024-05-08T01:38:06.344 INFO:teuthology.task.clock:Checking final clock skew...
2024-05-08T01:38:06.345 DEBUG:teuthology.orchestra.run.smithi106:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true
2024-05-08T01:38:06.347 DEBUG:teuthology.orchestra.run.smithi142:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true
2024-05-08T01:38:06.356 INFO:teuthology.orchestra.run.smithi106.stdout:     remote           refid      st t when poll reach   delay   offset  jitter
2024-05-08T01:38:06.356 INFO:teuthology.orchestra.run.smithi106.stdout:==============================================================================
2024-05-08T01:38:06.356 INFO:teuthology.orchestra.run.smithi106.stdout:+hv01.front.sepi 67.205.162.81    3 u   54   64  377    0.071   -9.404   2.101
2024-05-08T01:38:06.356 INFO:teuthology.orchestra.run.smithi106.stdout:*hv02.front.sepi 63.231.80.2      3 u   45   64  377    0.072   -8.380   1.249
2024-05-08T01:38:06.357 INFO:teuthology.orchestra.run.smithi106.stdout:+hv03.front.sepi 74.6.168.72      3 u   52   64  377    0.106   -6.439   1.790
2024-05-08T01:38:06.357 INFO:teuthology.orchestra.run.smithi106.stdout: hv04.front.sepi .INIT.          16 u    - 1024    0    0.000   +0.000   0.000
2024-05-08T01:38:06.362 INFO:teuthology.orchestra.run.smithi142.stdout:     remote           refid      st t when poll reach   delay   offset  jitter
2024-05-08T01:38:06.362 INFO:teuthology.orchestra.run.smithi142.stdout:==============================================================================
2024-05-08T01:38:06.362 INFO:teuthology.orchestra.run.smithi142.stdout:*hv01.front.sepi 67.205.162.81    3 u   54   64  377    0.088   -4.887   0.778
2024-05-08T01:38:06.362 INFO:teuthology.orchestra.run.smithi142.stdout:+hv02.front.sepi 63.231.80.2      3 u   55   64  377    0.101   -5.005   0.735
2024-05-08T01:38:06.363 INFO:teuthology.orchestra.run.smithi142.stdout:+hv03.front.sepi 74.6.168.72      3 u   65   64  377    0.098   -5.212   0.765
2024-05-08T01:38:06.363 INFO:teuthology.orchestra.run.smithi142.stdout: hv04.front.sepi .INIT.          16 u    - 1024    0    0.000   +0.000   0.000
2024-05-08T01:38:06.363 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab
2024-05-08T01:38:06.373 INFO:teuthology.task.ansible:Skipping ansible cleanup...
2024-05-08T01:38:06.373 DEBUG:teuthology.run_tasks:Unwinding manager selinux
2024-05-08T01:38:06.382 DEBUG:teuthology.run_tasks:Unwinding manager pcp
2024-05-08T01:38:06.391 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer
2024-05-08T01:38:06.400 INFO:teuthology.task.internal:Duration was 2056.099457 seconds
2024-05-08T01:38:06.400 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog
2024-05-08T01:38:06.409 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring...
2024-05-08T01:38:06.410 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart
2024-05-08T01:38:06.412 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart
2024-05-08T01:38:06.459 INFO:teuthology.task.internal.syslog:Checking logs for errors...
2024-05-08T01:38:06.460 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi106.front.sepia.ceph.com
2024-05-08T01:38:06.460 DEBUG:teuthology.orchestra.run.smithi106:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1
2024-05-08T01:38:06.500 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi142.front.sepia.ceph.com
2024-05-08T01:38:06.500 DEBUG:teuthology.orchestra.run.smithi142:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1
2024-05-08T01:38:06.513 INFO:teuthology.task.internal.syslog:Compressing syslogs...
2024-05-08T01:38:06.513 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
2024-05-08T01:38:06.544 DEBUG:teuthology.orchestra.run.smithi142:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
2024-05-08T01:38:06.583 INFO:teuthology.task.internal.syslog:Gathering journactl -b0...
2024-05-08T01:38:06.583 DEBUG:teuthology.orchestra.run.smithi106:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz
2024-05-08T01:38:06.612 DEBUG:teuthology.orchestra.run.smithi142:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz
2024-05-08T01:38:06.682 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo
2024-05-08T01:38:06.693 INFO:teuthology.task.internal:Restoring /etc/sudoers...
2024-05-08T01:38:06.693 DEBUG:teuthology.orchestra.run.smithi106:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers
2024-05-08T01:38:06.704 DEBUG:teuthology.orchestra.run.smithi142:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers
2024-05-08T01:38:06.718 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump
2024-05-08T01:38:06.728 DEBUG:teuthology.orchestra.run.smithi106:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
2024-05-08T01:38:06.752 DEBUG:teuthology.orchestra.run.smithi142:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
2024-05-08T01:38:06.759 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = core
2024-05-08T01:38:06.770 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = core
2024-05-08T01:38:06.782 DEBUG:teuthology.orchestra.run.smithi106:> test -e /home/ubuntu/cephtest/archive/coredump
2024-05-08T01:38:06.826 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T01:38:06.827 DEBUG:teuthology.orchestra.run.smithi142:> test -e /home/ubuntu/cephtest/archive/coredump
2024-05-08T01:38:06.832 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T01:38:06.832 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive
2024-05-08T01:38:06.843 INFO:teuthology.task.internal:Transferring archived files...
2024-05-08T01:38:06.843 DEBUG:teuthology.misc:Transferring archived files from smithi106:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696665/remote/smithi106
2024-05-08T01:38:06.844 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- .
2024-05-08T01:38:06.892 DEBUG:teuthology.misc:Transferring archived files from smithi142:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696665/remote/smithi142
2024-05-08T01:38:06.893 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- .
2024-05-08T01:38:06.914 INFO:teuthology.task.internal:Removing archive directory...
2024-05-08T01:38:06.914 DEBUG:teuthology.orchestra.run.smithi106:> rm -rf -- /home/ubuntu/cephtest/archive
2024-05-08T01:38:06.932 DEBUG:teuthology.orchestra.run.smithi142:> rm -rf -- /home/ubuntu/cephtest/archive
2024-05-08T01:38:06.958 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload
2024-05-08T01:38:06.970 INFO:teuthology.task.internal:Not uploading archives.
2024-05-08T01:38:06.970 DEBUG:teuthology.run_tasks:Unwinding manager internal.base
2024-05-08T01:38:06.979 INFO:teuthology.task.internal:Tidying up after the test...
2024-05-08T01:38:06.980 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2024-05-08T01:38:06.982 DEBUG:teuthology.orchestra.run.smithi142:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2024-05-08T01:38:06.986 INFO:teuthology.orchestra.run.smithi106.stdout:   658636      4 drwxr-xr-x   2 ubuntu   ubuntu       4096 May  8 01:38 /home/ubuntu/cephtest
2024-05-08T01:38:07.006 INFO:teuthology.orchestra.run.smithi142.stdout:   658748      4 drwxr-xr-x   2 ubuntu   ubuntu       4096 May  8 01:38 /home/ubuntu/cephtest
2024-05-08T01:38:07.007 DEBUG:teuthology.run_tasks:Unwinding manager kernel
2024-05-08T01:38:07.018 DEBUG:teuthology.run_tasks:Unwinding manager console_log
2024-05-08T01:38:07.039 INFO:teuthology.run:Summary data:
description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node
  openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid
  supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-policy-none}
duration: 2056.0994567871094
flavor: default
owner: scheduled_dis@teuthology
success: true

2024-05-08T01:38:07.040 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-05-08T01:38:07.123 INFO:teuthology.run:pass